Method for fast, robust, multi-dimensional pattern recognition

ABSTRACT

Disclosed is a method for determining the absence or presence of one or more instances of a predetermined pattern in an image, and for determining the location of each found instance within a multidimensional space. A model represents the pattern to be found, the model including a plurality of probes. Each probe represents a relative position at which a test is performed in an image at a given pose, each such test contributing evidence that the pattern exists at the pose. The method further includes a comparison of the model with a run-time image at each of a plurality of poses. A match score is computed at each pose to provide a match score surface. Then, the match score is compared with an accept threshold, and used to provide the location any instances of the pattern in the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of application Ser. No. 09/114,335, Filed Jul.13, 1998, now U.S. Pat. No. 7,016,539 the contents of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

Digital images are formed by many devices and used for many practicalpurposes. Devices include TV cameras operating on visible or infraredlight, line-scan sensors, flying spot scanners, electron microscopes,X-ray devices including CT scanners, magnetic resonance imagers, andother devices known to those skilled in the art. Practical applicationsare found in industrial automation, medical diagnosis, satellite imagingfor a variety of military, civilian, and scientific purposes,photographic processing, surveillance and traffic monitoring, documentprocessing, and many others.

To serve these applications the images formed by the various devices areanalyzed by digital devices to extract appropriate information. One formof analysis that is of considerable practical importance is determiningthe position, orientation, and size of patterns in an image thatcorrespond to objects in the field of view of the imaging device.Pattern location methods are of particular importance in industrialautomation, where they are used to guide robots and other automationequipment in semiconductor manufacturing, electronics assembly,pharmaceuticals, food processing, consumer goods manufacturing, and manyothers.

Another form of digital image analysis of practical importance isidentifying differences between an image of an object and a storedpattern that represents the “ideal” appearance of the object. Methodsfor identifying these differences are generally referred to as patterninspection methods, and are used in industrial automation for assembly,packaging, quality control, and many other purposes.

One early, widely-used method for pattern location and inspection isknown as blob analysis. In this method, the pixels of a digital imageare classified as “object” or “background” by some means, typically bycomparing pixel gray-levels to a threshold. Pixels classified as objectare grouped into blobs using the rule that two object pixels are part ofthe same blob if they are neighbors; this is known as connectivityanalysis. For each such blob one determines properties such as area,perimeter, center of mass, principal moments of inertia, and principalaxes of inertia. The position, orientation, and size of a blob is takento be its center of mass, angle of first principal axis of inertia, andarea, respectively. These and the other blob properties can be comparedagainst a known ideal for proposes of inspection.

Blob analysis is relatively inexpensive to compute, allowing for fastoperation on inexpensive hardware. It is reasonably accurate under idealconditions, and well-suited to objects whose orientation and size aresubject to change. One limitation is that accuracy can be severelydegraded if some of the object is missing or occluded, or if unexpectedextra features are present.

Another limitation is that the values available for inspection purposesrepresent coarse features of the object, and cannot be used to detectfine variations. The most severe limitation, however, is that exceptunder limited and well-controlled conditions there is in general noreliable method for classifying pixels as object or background. Theselimitations forced developers to seek other methods for pattern locationand inspection.

Another method that achieved early widespread use is binary templatematching. In this method a training image is used that contains anexample of the pattern to be located. The subset of the training imagecontaining the example is thresholded to produce a binary pattern andthen stored in a memory. At run-time, images are presented that containthe object to be found. The stored pattern is compared with like-sizedsubsets of the run-time image at all or selected positions, and theposition that best matches the stored pattern is considered the positionof the object. Degree of match at a given position of the pattern issimply the fraction of pattern pixels that match their correspondingimage pixel.

Binary template matching does not depend on classifying image pixels asobject or background, and so it can be applied to a much wider varietyof problems than blob analysis. It also is much better able to toleratemissing or extra pattern features without severe loss of accuracy, andit is able to detect finer differences between the pattern and theobject. One limitation, however, is that a binarization threshold isneeded, which can be difficult to choose reliably in practice,particularly under conditions of poor signal-to-noise ratio or whenillumination intensity or object contrast is subject to variation.Accuracy is typically limited to about one whole pixel due to thesubstantial loss of information associated with thresholding. Even moreserious, however, is that binary template matching cannot measure objectorientation and size. Furthermore, accuracy degrades rapidly with smallvariations in orientation and/or size, and if larger variations areexpected the method cannot be used at all.

A significant improvement over binary template matching came with theadvent of relatively inexpensive methods for the use of gray-levelnormalized correlation for pattern location and inspection. Thesemethods are similar to binary template matching, except that nothreshold is used so that the full range of image gray-levels areconsidered, and the degree of match becomes the correlation coefficientbetween the stored pattern and the image subset at a given position.

Since no binarization threshold is needed, and given the fundamentalnoise immunity of correlation, performance is not significantlycompromised under conditions of poor signal-to-noise ratio or whenillumination intensity or object contrast is subject to variation.Furthermore, since there is no loss of information due to thresholding,position accuracy down to about ¼ pixel is practical using well-knowninterpolation methods. The situation regarding orientation and size,however, is not much improved.

Another limitation of correlation methods is that in many applicationsobject shading can vary locally and non-linearly across an object,resulting in poor correlation with the stored pattern and thereforefailure to locate it. For example, in semiconductor fabrication theprocess step known as chemical mechanical planarization (CMP) results inradical, non-linear changes in pattern shading, which makes alignmentusing correlation impossible. As another example, in almost anyapplication involving 3-dimensional objects, such as robotpick-and-place applications, shading will vary as a result of variationsin angles of illumination incidence and reflection, and from shadows andmutual illumination. The effects are more severe for objects thatexhibit significant specular reflection, particularly metals andplastics.

More recently, improvements to gray-level correlation have beendeveloped that allow it to be used in applications where significantvariation in orientation and/or size is expected. In these methods, thestored pattern is rotated and/or scaled by digital image re-samplingmethods before being matched against the image. By matching over a rangeof angles, sizes, and x-y positions, one can locate an object in thecorresponding multidimensional space. Note that such methods would notwork well with binary template matching, due to the much more severepixel quantization errors associated with binary images.

One problem with these methods is the severe computational cost, both ofdigital re-sampling and of searching a space with more than 2dimensions. To manage this cost, the search methods break up the probleminto two or more phases. The earliest phase uses a coarse, subsampledversion of the pattern to cover the entire search space quickly andidentify possible object locations. Subsequent phases use finer versionsof the pattern to refine the positions determined at earlier phases, andeliminate positions that the finer resolution reveals are not wellcorrelated with the pattern. Note that variations of these coarse-finemethods have also been used with binary template matching and theoriginal two-dimensional correlation, but are even more important withthe higher-dimensional search space.

Even with these techniques, however, the computational cost is stillhigh, and the problems associated with non-linear variation in shadingremain.

Another pattern location method in common use is known as theGeneralized Hough Transform (GHT). This method traces its origins toU.S. Pat. No. 3,069,654 [Hough, P. V. C., 1962], which described amethod for locating parameterized curves such as lines or conicsections. Subsequently the method was generalized to be able to locateessentially arbitrary patterns. As with the above template matching andcorrelation methods, the method is based on a trained pattern. Insteadof using gray levels directly, however, the GHT method identifies pointsalong object boundaries using well-known methods of edge detection. Alarge array of accumulators, called Hough space, is constructed, withone such accumulator for each position in the multidimensional space tobe searched. Each edge point in the image corresponds to a surface ofpossible pattern positions in Hough space. For each such edge point, theaccumulators along the corresponding surface are incremented. After allimage edge points have been processed, the accumulator with the highestcount is considered to be the multidimensional location of the pattern.

The general performance characteristics of GHT are very similar tocorrelation. Computational cost rises very rapidly with number ofdimensions, and although coarse-fine methods have been developed toimprove performance, practical applications beyond 2 dimensions arealmost nonexistent.

The edge detection step of GHT generally reduces problems due tonon-linear variations in object contrast, but introduces new problems.Use of edge detectors generally increases susceptibility to noise anddefocus. For many objects the edges are not sharply defined enough forthe edge detection step to yield reliable results. Furthermore, edgedetection fundamentally requires a binarization step, where pixels areclassified as “edge” or “not edge”, usually by a combination ofthresholding and peak detection. Binarization, no matter what method isused, is always subject to uncertainty and misclassification, and willcontribute failure modes to any method that requires it.

Terminology

The following terminology is used throughout the specification:

-   -   Object—Any physical or simulated object, or portion thereof,        having characteristics that can be measured by an image forming        device or simulated by a data processing device.    -   Image—A 2-dimensional function whose values correspond to        physical characteristics of an object, such as brightness        (radiant energy, reflected or otherwise), color, temperature,        height above a reference plane, etc., and measured by any        image-forming device, or whose values correspond to simulated        characteristics of an object, and generated by any data        processing device.    -   Brightness—The physical or simulated quantity represented by the        values of an image, regardless of source.    -   Granularity—A selectable size (in units of distance) below which        spatial variations in image brightness are increasingly        attenuated, and below which therefore image features        increasingly cannot be resolved. Granularity can be thought of        as being related to resolution.    -   Boundary—An imaginary contour, open-ended or closed, straight or        curved, smooth or sharp, along which a discontinuity of image        brightness occurs at a specified granularity, the direction of        said discontinuity being normal to the boundary at each point.    -   Gradient—A vector at a given point in an image giving the        direction and magnitude of greatest change in brightness at a        specified granularity at said point.    -   Pattern—A specific geometric arrangement of contours lying in a        bounded subset of the plane of the contours, said contours        representing the boundaries of an idealized image of an object        to be located and/or inspected.    -   Model—A set of data encoding characteristics of a pattern to be        found for use by a pattern finding method.    -   Training—The act of creating a model from an image of an example        object or from a geometric description of an object or a        pattern.    -   Pose—A mapping from pattern to image coordinates and        representing a specific transformation and superposition of a        pattern onto an image.

SUMMARY OF THE INVENTION

In one aspect the invention is a general-purpose method for determiningthe absence or presence of one or more instances of a predeterminedpattern in an image, and determining the location of each foundinstance. The process of locating patterns occurs within amultidimensional space that can include, but is not limited to, x-yposition (also called translation), orientation, and size. In anotheraspect the invention is a method for identifying differences between apredetermined pattern and a matching image subset. The process ofidentifying differences is called inspection.

To avoid ambiguity we will call the location of a pattern in amultidimensional space its pose. More precisely, a pose is a coordinatetransform that maps points in a pattern to corresponding points in animage. In a preferred embodiment, a pose is a general 6 degree offreedom linear coordinate transform. The 6 degrees of freedom can berepresented by the 4 elements of a 2×2 matrix, plus the 2 elements of avector corresponding to the 2 translation degrees of freedom.Alternatively and equivalently, the 4 non-translation degrees of freedomcan be represented in other ways, such as orientation, size, aspectratio, and shear, or x-size, y-size, x-axis-angle, and y-axis-angle.

The results produced by the invention can be used directly, or can befurther refined by multidimensional localization methods such asdescribed in U.S. Pat. No. 6,658,145 entitled “Fast High-AccuracyMulti-Dimensional Pattern Inspection”.

The invention uses a model that represents the pattern to be found. Themodel can be created from a training image or synthesized from ageometric description. The invention is a template matching method—themodel is compared to an image at each of an appropriate set of poses, amatch score is computed at each pose, and those poses that correspond toa local maximum in match score, and whose match scores are above asuitable accept threshold, are considered instances of the pattern inthe image.

According to the invention, a model includes a set of data elementscalled probes. Each probe represents a relative position at whichcertain measurements and tests are to be made in an image at a givenpose, each such test contributing evidence that the pattern exists atsaid pose. In one embodiment of the invention, each probe represents ameasurement and test of gradient direction. In another embodiment, eachprobe represents a measurement and test of both gradient direction andmagnitude. In a preferred embodiment, the probes represent differenttests at different steps of the method. The gradient magnitude ordirection to be tested by a probe is referred to as the gradientmagnitude or direction under the probe.

In a preferred embodiment, a probe is defined by its position,direction, and weight. Each of these quantities are conceptually realnumbers, although of course in any actual embodiment they would berepresented as floating or fixed point approximations. Probe position isa point in a pattern coordinate system at which, after transforming to aimage coordinate system using a given pose, a measurement and test is tobe made. Probe direction is the expected gradient direction in patterncoordinates at the indicated position, which also must be transformed toimage coordinates prior to use. Probe weight gives the relativeimportance of the probe in determining the presence and location of thepattern.

In a preferred embodiment, probe weights can be positive or negative. Anegative weight indicates that a test showing similar gradient directionand sufficient gradient magnitude should count as evidence against theexistence of the pattern at the specified pose.

Most points in an image contain little useful information about patternposition. Uniform regions, for example, contain no information aboutposition, since brightness is locally independent of position. Generallythe second or higher derivative of brightness must be non-zero in somedirection for there to be useful information, and it has long beenrecognized in the art that the best information occurs along boundaries.Thus examining an image at every point for the purpose of patternlocation is unnecessary as well as wasteful of memory and processingtime.

In a preferred embodiment, a model includes a small set of probes placedat selected points along the boundaries represented by the correspondingpattern. The probes are uniformly spaced along segments of theboundaries characterized by a small curvature. The spacing between theprobes is chosen so that a predetermined number of probes is used,except that fewer probes can be used to prevent the spacing from beingset below some predetermined minimum value, and more probes can be usedto prevent the spacing from being set above some predetermined maximumvalue. In a preferred embodiment, the said predetermined number ofprobes is 64.

The boundaries that appear in a given image are not absolute but dependon the granularity at which the image is interpreted. Consider forexample a newspaper photograph. Over some range of very finegranularities, one perceives nothing but a pattern of dots of varioussizes and separations. Over some range of coarser granularity, the dotscannot be resolved and one may perceive human facial features such aseyes, noses, and mouths. At even coarser granularity, one may perceiveonly human heads.

For an image sensor producing a digital image, granularity is limited bypixel size and sharpness of focus. Granularity may be increased abovethis limit (i.e. made coarser) by suitable image processing operations,and thus effectively controlled over a wide range. In a pattern locatingsystem, choice of granularity affects speed, accuracy, and reliability.When suitable methods are used, pattern locating speed can be made toincrease rapidly as granularity increases, which can be crucial for highspeed applications where the pattern's pose can vary in more than 2degrees of freedom. Pattern location accuracy, however, decreases asgranularity increases. Pattern locating reliability, the ability tocorrectly identify patterns when they exist and to avoid misidentifyingimage subsets that are not instances of the pattern, may fall off if thegranularity is too coarse to resolve key pattern features, and may falloff if the granularity is so fine that details are resolved that areinconsistent from instance to instance, such as surface texture or otherrandom microstructure.

In a preferred embodiment of the invention, granularity is selectableover a wide range down to the limit imposed by the image sensor. Inanother preferred embodiment, a suitable granularity is automaticallychosen during model training. In another preferred embodiment, at leasttwo granularities are used, so that the speed advantages of the coarsestgranularity and the accuracy advantages of the finest granularity can beobtained. In the preferred embodiment wherein at least two granularitiesare used, the model includes a separate set of probes for eachgranularity.

Granularity can be increased above the image sensor limit by a low-passfiltering operation, optionally followed by a sub-sampling operation.Methods for low-pass filtering and subsampling of digital images arewell known in the art. Until recently, however, inexpensive, high speedmethods that could be tuned over a wide range with no significant lossin performance were not available. In a preferred embodiment, theinvention makes use of a constant-time second-order approximatelyparabolic filter, as described in U.S. Pat. No. 6,457,032, entitled“Efficient, Flexible Digital Filtering”, followed by a non-integersub-sampling step wherein brightness values spaced g pixels apart,horizontally and vertically, are linearly interpolated between thefiltered pixel values for some value of g chosen at training time.

Methods for estimating image gradient magnitude and direction are wellknown in the art, but most methods in common use are either too slow orof insufficient accuracy to be suitable for the practice of theinvention. For example, most commercially available gradient estimationmethods can only resolve direction to within 45°, and only provide acrude estimate of magnitude. One notable exception is described in U.S.Pat. No. 5,657,403, herein incorporated by reference, although at thetime of that patent specialized hardware was required for high speedoperation. Recent advances in computer architecture and performance havemade high speed, accurate gradient estimation practical on inexpensivehardware. In a preferred embodiment, the invention uses the well-knownSobel kernels to estimate the horizontal and vertical components ofgradient, and the well-known CORDIC algorithm, as described, forexample, in U.S. Pat. No. 6,408,109, entitled “Apparatus and Method forDetecting and Sub-Pixel Location of Edges in a Digital Image”, hereinincorporated by reference, for example, to compute gradient magnitudeand direction. The Sobel kernels are applied either to the input imageor to a filtered, sub-sampled image, as described above, so that theresult is a gradient magnitude image and a gradient direction image thattogether provide image gradient information at uniformly spaced points,which mayor may not correspond to the pixels of the input image, and ata selectable granularity. In a preferred embodiment, the gradientmagnitude and direction images are stored in a random access memory of acomputer or other data processing device, in such a manner that theaddress difference between pixels in the horizontal direction is a firstconstant, and the address difference between pixels in the verticaldirection is a second constant.

The method of the invention, which tests gradient direction at each of asmall set (e.g. 64) of positions, offers many advantages over prior artmethods of template matching. Since neither probe position nor directionare restricted to a discrete pixel grid, and since probes representpurely geometric information and not image brightness, they can betranslated, rotated, and scaled much faster than digital imagere-sampling methods and with less pixel grid quantization error.Furthermore, since probes are spaced along contours where a maximumamount of position information occurs, a small set of probes can be usedso that processing time can be minimized.

Gradient direction is a much more reliable basis for pattern locationthan image brightness. Brightness may vary as a result of object surfacereflectance, intensity of illumination, angles of illumination incidenceand reflection, mutual illumination and shadows, sensor gain, and otherfactors. Gradient direction at boundaries is generally unaffected bythese factors as long as the overall shape of the object is reasonablyconsistent. Furthermore, each individual test of gradient directionprovides direct evidence of the presence or absence of a pattern at agiven pose, and this evidence has absolute meaning that is independentof the conditions under which a given image was obtained. For example,one generally can conclude that a direction error of 3 degrees is a goodmatch, and of 30 degrees is a poor match, without knowing anything aboutthe pattern to be located or any of the above listed factors affectingthe individual brightness values in any given image. By contrast, a testof image brightness is meaningless in itself-whether 3 brightness unitsof difference or 30 units of difference is good or bad can only beassessed in relation to the statistics of a large set of brightnessvalues.

The method of the invention also offers many advantages over prior artmethods based on the Hough transform. The high quality of theinformation provided by tests of gradient direction allows fewer pointsto be processed, resulting in higher speed. Sets of probes can berotated and scaled more quickly and accurately than the edge point setsused by GHT methods. Hough methods including the GHT tend to beadversely affected by small variations in pattern shape, or edgeposition quantization error, where a shift in edge position by even onepixel will cause an undesirable spreading of the peak in Hough space. Bycontrast, gradient direction is generally consistent within a couple ofpixels of a boundary, so that the effects of small variations in shapeor quantization errors are generally insignificant. Loss of sharp focuscan degrade the edge detection step required for Hough methods, whereasdefocus has no effect on gradient direction. Hough transform methods,and all methods based on edge detection, fundamentally require abinarization step, where pixels are classified as “edge” or “not edge”,and all such methods are subject to uncertainty and misclassification.The use of gradient direction by the invention requires no binarizationor other classification to be applied.

A variety of match functions based on gradient direction, and optionallygradient magnitude and probe weight, can be used within the scope of theinvention. In a first match function, probe positions having gradientdirection errors below a first predetermined value are given a rating of1.0, above a second predetermined value are given a rating of 0, anderrors that fall between the said first and second values are given arating proportionally between 0 and 1.0. The weighted sum of proberatings, divided by the total weight of all probes, is the match score.With said first match function all probe weights are positive, since anegative weight probe cannot be considered to provide evidence against apose unless the gradient magnitude is tested and found to besufficiently strong. The first match function results in the highestpossible speed, for two primary reasons. First, gradient magnitude isnot used, which reduces both the calculations needed and the number ofaccesses to memory wherein gradient magnitude information would bestored. Second, due to the general consistency of gradient directionsurrounding a boundary, the first match function tends to produce broadpeaks in match score, which allows a relatively sparse set of poses tobe evaluated.

In a first variation on said first match function probe weights are notused (i.e. probe weights are effectively all 1.0), which furtherincreases the speed of operation. In a second variation on said firstmatch function, the expected value of the weighted sum of the proberatings on random gradient directions is subtracted from the actualweighted sum, with the total weight adjusted accordingly, so that aperfect match still gets a score of 1.0 but the expected value of thescore on random noise is 0.

In a second match function, a direction rating factor is computed foreach probe that is the same as the probe rating used by the first matchfunction, and probes receive a rating that is the product of thedirection rating factor and the gradient magnitude under the probe. Thematch score is the weighted sum of the probe ratings. With the secondmatch function, probe weights can be positive or negative. The secondmatch function produces sharper peaks in match score than the first,since gradient magnitude is at a maximum at a boundary and falls offsharply on either side. As a result pattern position can be determinedmore accurately with the second match function, but at a cost of lowerspeed since more calculations and memory accesses are needed and since adenser set of poses must be evaluated. Unlike the first match functionwhich produces a score between 0 and 1.0, the second match function'sscore is essentially open-ended and dependent on boundary contrast. Thuswhile the score can be used to compare a pose to a neighboring pose todetermine a peak in match score, it cannot in general be used to comparea pose to a distant pose or to provide a value that can be used reliablyto judge whether or not an instance of the pattern is present in theimage at a given pose.

In a third match function, a direction rating factor is computed foreach probe identical to that of the second match function, and amagnitude rating factor is computed that is 1.0 for gradient magnitudesabove a certain first value, 0 for magnitudes below a certain secondvalue, and proportionally between 0 and 1.0 for values between saidfirst and second values. Each probe receives a rating that is theproduct of the direction rating factor and the magnitude rating factor,and the match score is the weighted sum of the probe ratings divided bythe total weight of all the probes. Probe weights can be positive ornegative. In a preferred embodiment, the said first value is computedbased on image characteristics at any pose for which the third matchfunction is to be evaluated, so the third match function takes thelongest to compute. Furthermore, peaks in match score are generally lesssharp than for the second match function, so position is less accurate.The primary advantage of the third match function is that it produces ascore that falls between 0 and 1.0 that can be used for comparison andto judge whether or not an instance of the pattern is present in theimage at a given pose, and that said score takes into account gradientmagnitude and allows negative probe weights.

In the aspect of the invention where inspection is to be performed, thescore produced by the third match function is used to provide an overallmeasure of the quality of a specific instance of the pattern found in animage, and the individual probe ratings computed during evaluation ofthe third match function are used to provide more detailed informationabout differences between the found instance and the pattern.

In a preferred embodiment, the first match function, and using both thefirst and second variation, is used during a coarse scan step duringwhich the entire multidimensional search space is evaluated with arelatively sparse set of poses. Poses that are coarse peaks,specifically those at which the first match score is a local maximum andabove a predetermined accept threshold, are refined during a fine scanstep that evaluates a small, dense set of poses surrounding each coarsepeak. The fine scan step uses the second match function to achieve aprecise position and to consider the evidence of negative weight probes.An interpolation between the pose resulting in the highest value of thesecond match function and its neighbors is considered the location ofone potential instance of the pattern in the image. A scoring stepevaluates the third match function at this final, interpolated pose tojudge whether or not an instance of the pattern is actually present inthe image at said pose by comparing the value of the third matchfunction to an accept threshold.

In any specific embodiment of the invention the search space is definedby certain degrees of freedom that include the two translation degreesof freedom and some number, possibly zero, of non-translation degrees offreedom such as orientation and size. Many methods can be devised withinthe scope of the invention to generate the set of poses to be evaluatedfor purposes of pattern location. In a preferred embodiment, anyspecific pose is the result of specifying values for each degree offreedom. The set of poses to be evaluated during the coarse scan step isthe result of generating all combinations of selected values for eachdegree of freedom. For this preferred embodiment, two distinct methodsare used in combination to generate the set of poses, one fortranslation and one for non-translation degrees of freedom.

According to this preferred embodiment of the invention, for eachcombination of values of the non-translation degrees of freedom theprobe positions and directions are transformed according to the saidcombination of values from pattern coordinates to an image coordinatesystem associated with the gradient magnitude and direction images. Theresulting positions, which are relative positions since the translationdegrees of freedom have not yet been included, are rounded to relativeinteger pixel coordinates and, using the horizontal and vertical addressdifference constants, converted to a single integer offset value thatgives the relative position of the probe in either the gradientmagnitude or direction image at poses corresponding to the saidcombination of non-translation degrees of freedom. The result is a newset of data elements called compiled probes that include relative imageoffset, transformed expected gradient direction, and weight.

According to this preferred embodiment of the invention, during thecoarse step the compiled probes are used to evaluate the first matchfunction at a set of translations corresponding to some regulartessellation. For any such translation that is a local maximum in firstmatch score, and where said score is above a suitable accept threshold,a set of data called a result is allocated and added to a list ofresults. A translation is interpolated between the local maximum and itsneighbors and stored in the newly-allocated result, along with aninterpolated score. According to this preferred embodiment of theinvention, a hexagonal tessellation is used along with methods fordetermining a local maximum and for interpolation on such atessellation. In a less preferred variation on this embodiment, aconventional square tessellation is used, including well-known methodsfor determining the presence of a local maximum and for interpolatingbetween said maximum and its neighbors.

According to this preferred embodiment of the invention, eachnon-translation degree of freedom is defined and described by a set ofdata and functions called a generalized-DOF. Each generalized-DOFincludes a single real-valued (or floating point approximation)parameter that specifies its value, for example an orientation degree offreedom would have an angle parameter and a size degree of freedom wouldhave a scale factor parameter. Each generalized-DOF includes a functionthat maps the parameter value to a corresponding 2-dimensionalcoordinate transform. For example, for an orientation generalized-DOFthis function might include the matrix

$\begin{matrix}\begin{pmatrix}{\cos(x)} & {- {\sin(x)}} \\{\sin(x)} & {\cos(x)}\end{pmatrix} & (1)\end{matrix}$where “x” is the parameter. Each generalized-DOF includes a low and highlimit value that specifies the range of parameter values within whichthe pattern should be located for the degree of freedom, and which areset based on the requirements of the application to which the inventionis being used. Each generalized-DOF includes a value that if non-zerospecifies the period of cyclic degrees of freedom such as orientation(e.g., 360 if the parameter is in degrees). Each generalized-DOFincludes a maximum step size value that specifies the maximum allowableseparation between parameter values used to generate poses for thedegree of freedom for the coarse scan step. In the preferred embodiment,the maximum step size for each generalized-DOF is determined from ananalysis of the magnitude of motion of the probes in the expectedgradient direction as the generalized-DOF parameter is varied. In a lesspreferred embodiment, the maximum step size is a fixed, predeterminedvalue.

According to this preferred embodiment of the invention, for a givengeneralized-DOF if the difference between the high and low limit values,which is referred to as the parameter range, is not greater than themaximum step size, then the generalized-DOF parameter is not variedduring the coarse step, but is instead set to the halfway point betweenthe limits. If the parameter range is greater than the maximum stepsize, then an actual step size is computed such that the actual stepsize is not greater than the maximum and the range is an integermultiple of the actual step size. For the given generalized-DOF, the setof parameter values generated for the coarse scan step range fromone-half of the actual step size below the low limit to one-half of theactual step size above the high limit, in increments of the actual stepsize. It can be seen that a minimum of three distinct parameter valuesare generated in this case.

The invention uses a set of nested loops during the coarse scan step togenerate all combinations of parameter values of the generalized-DOFs inuse, where each such loop corresponds to one generalized-DOF. Each loopsteps the parameter value of the corresponding generalized-DOF over therange and using the actual step size as described above, generating acoordinate transform corresponding to each parameter value. In theinnermost loop, the coordinate transforms corresponding to the currentparameter values of all of the generalized-DOFs are composed to producea single overall transform specifying all of the non-translation degreesof freedom as needed for the translation degrees of freedom as describedabove. Data specifying the values of the generalized-DOF parameters areadded to the results lists produced during scanning of the translationdegrees of freedom. At the end of each of the nested loops, the resultslists are scanned to identify sets of results that correspond to thesame instance of the pattern in the image at a consecutive sequence ofparameter values of the generalized-DOF corresponding to the givennested loop. For each such set found, all but the peak result (the onewith the highest score) are deleted, and the parameter value and scoreare interpolated between the peak result and its neighbors. All of theremaining (i.e. peak) results are concatenated to produce a singlemaster results list representing the results of the coarse scan step.

During the fine scan step the pose of each result produced by the coarsescan step is refined one or more times. For each such refinement, all ofthe translation and non-translation degrees of freedom are analyzed andupdated, using a revised actual step size for the generalized-DOFs thatis one-half that of the previous refinement step. For the firstrefinement step, the revised actual step size for the generalized-DOFsis one-half that of the coarse scan step. As for the coarse scan step,two distinct methods are used in combination to generate a set of poses,one for translation and one for non-translation degrees of freedom.

For the translation degrees of freedom, compiled probes are generated asbefore using the composed coordinate transform that specifies the valueall of the non-translation degrees of freedom. The second match functionis evaluated at every pixel offset within a small, approximatelycircular region centered on the translation stored in the result beingrefined. A new translation is stored that is interpolated between themaximum value of the second match function and its neighbors.

The invention uses a set of nested loops during the fine scan step togenerate certain combinations of parameter values of thegeneralized-DOFs in use, where each such loop corresponds to onegeneralized-DOF. For each such loop three parameter values are chosen tostart—the current value stored in the result, a value lower by thecurrent revised actual step size, and a value higher by the currentrevised actual step size. If the lower parameter value results in asecond match score that is higher than the match scores resulting fromthe other two parameter values, further lower parameter values aregenerated in steps of the current revised actual step size, until eithera parameter value is found that does not result in a higher second matchscore than the other parameter values, or the lower limit for thisgeneralized-DOF is reached. If the lower limit is reached, the finalparameter value is stored in the result for this generalized-DOF. Ifinstead a peak second match value was found, the parameter value isinterpolated between said peak and its neighbors and stored in theresult. Similar steps are followed if at the start of the nested loopthe higher parameter value results in a second match score that ishigher than the match scores resulting from the other two parametervalues.

Sometimes two or more results produced by the coarse scan step areduplicates that correspond to the same instance of the pattern in theimage, differing only slightly in pose. ill a preferred embodiment ofthe invention, when results are found that overlap by more than somepredetermined amount in each degree of freedom, the result with thehighest score is kept and the other duplicates are deleted.

The steps associated with the generalized-DOFs for the coarse and finescan steps can be realized by means of a computer program. Followingconventional practice the nested loops described above can be codeddirectly based on the attributes of a predetermined set ofgeneralized-DOFs. Such a conventional method of coding, however, resultsin duplication for each generalized-DOF of substantially similar codefor both the coarse and fine scan steps, as well as other steps such asthe determining of the maximum and actual step sizes, since the stepsrequired for each generalized-DOF are basically the same with only minordifferences in how the parameter value is used to produce a coordinatetransform and whether or not the generalized-DOF is cyclic. Duplicationwith minor modifications of a significant amount of complex code foreach generalized-DOF results in a computer program where debugging,modification, and maintenance are difficult and error-prone. Adding orremoving specific generalized-DOFs, or changing the nesting order, wouldbe particularly difficult and error-prone and could only be done atcompile time.

One aspect of the invention is a solution to the problem of coding amethod for scanning non-translation degrees of freedom for the purposeof pattern locating. The coding method requires no duplication of code,and allows non-translation degrees of freedom to be added and removed,and to have the nesting order changed, at run time. The invention isbased on the so-called object oriented programming methodology that hasrecently become generally available in the form of programming languagessuch as C++ and Java, although the invention could be practiced usingmany conventional languages such as C and assembly language bysimulating the appropriate object oriented features.

In a preferred embodiment, each generalized-DOF is represented by a C++class that is derived from an abstract base class that specifies a fixedinterface to any generalized-DOF and implements all functionality thatis common to all generalized-DOFs. Any functionality that is specific toa generalized-DOF, including the function that maps the parameter valueto a coordinate transform, is specified as a virtual function in thebase class and overridden in each derived class for the specificgeneralized-DOF. Data that is specific to a generalized-DOF, such as theparameter limits, maximum and actual step sizes, and cyclic periodvalue, can be held in data members of the base class. Data specifyingfixed attributes of the generalized-DOF, such as the cyclic periodvalue, can be initialized by the constructor of the derived class.

According to this preferred embodiment, a list of generalized-DOFs(instances of the derived classes) is used to specify thenon-translation degrees of freedom to be scanned and the nesting order.The list can be constructed from available generalized-DOF classes atruntime. The nested loops of the coarse scan step are implemented by asingle non-virtual member function of the base class, which is given alist of generalized-DOFs as one argument. The coarse scan functionprocesses the generalized-DOF at the head of the list by generating theappropriate sequence of parameter values as described above, generatingthe corresponding coordinate transform, and then calling itselfrecursively on the remainder of the list of generalized-DOFs toimplement loops nested within the current one. When the coarse scanfunction is finally passed a null list of generalized-DOFs, it callsanother function to do the translation degrees of freedom. The nestedloops of the fine scan step are handled similarly by another non-virtualmember function of the base class.

The pattern location method of the invention is truly general purpose,because it has the following characteristics:

-   -   Essentially arbitrary patterns can be trained by example;    -   Variations in pattern location, orientation, and size can be        tolerated and measured;    -   Pattern defects such as missing or unexpected features can be        tolerated with no significant loss in accuracy;    -   Non-linear variations in shading can be tolerated;    -   Real-world imaging conditions such as defocus, video noise, and        illumination variation can be tolerated; and    -   Price/performance ratio is low enough for widespread use.

Prior to the present invention, there were no practical, trulygeneral-purpose pattern location methods available.

BRIEF DESCRIPTION OF THE DRAWING

The invention will be more fully understood from the following detaileddescription, in conjunction with the accompanying figures, wherein:

FIG. 1 is a high-level block diagram of one embodiment of the invention;

FIG. 2 is a is block diagram of the steps for granularity control,gradient estimation, and boundary detection;

FIG. 3 is a flow chart of a preferred embodiment of the model trainingstep of FIG. 1;

FIG. 4 is an illustration of an example of a portion of a boundary pointlist for a small subset of a training image;

FIGS. 5 a, 5 b, and 5 c show an illustration of a portion of FIG. 4,showing the details of the connect step of the model training step ofFIG. 3;

FIG. 6 is an illustration of the portion of FIG. 4, showing the detailsof the segment step of FIG. 3;

FIG. 7 is an illustration of a set of probes according to the inventionto be included in a model resulting from the training step of FIG. 1;

FIG. 8 is an illustration of a synthetic model according to theinvention for locating or inspecting a rounded rectangle pattern;

FIG. 9 is an illustration of various typographic and symbolic conventionused in the specification;

FIG. 10 is an illustration of a data set that represents a model of FIG.1;

FIG. 11 a is an illustration of a data set that represents a probe ofFIG. 10;

FIG. 11 b is an illustration of a data set that represents a compiledprobe generated from the probe of FIG. 11 a;

FIGS. 12 a and 12 b are flow charts illustrating the steps forconverting a list of probe objects into a list of compiled-probeobjects;

FIGS. 13 a and 13 b are illustrations of direction rating factorfunctions;

FIG. 13 c is an illustration of a magnitude rating factor function;

FIG. 14 is an illustration of a data set that represents ageneralized-DOF;

FIG. 15 is a table that details specific generalized-DOFs of FIG. 14that can be used with the invention;

FIG. 16 is an illustration of the details of a list of generalized-DOFsused in a preferred embodiment of the invention;

FIG. 17 is an illustration of a data set that represents a resultcorresponding to an instance of a pattern in a run-time image;

FIGS. 18 a, 18 b, and 18 c show an illustration of how position overlapis calculated for a pair of results to determine if they might beneighbors or duplicates;

FIG. 19 is a top-level flow chart of a preferred embodiment of run-time;

FIG. 20 is a flow chart of setting generalized-DOF elements;

FIGS. 21 a, 21 b, and 21 c show a flow chart of a function that scansall of the generalized-DOFs on an input list, and returns a list ofresults describing poses representing possible instance of a pattern ina run-time image;

FIGS. 22 a, and 22 b show a flow chart of a function used to scan thetranslation degrees of freedom;

FIG. 23 is an illustration of four different preferred coarse scanpatterns;

FIG. 24 a shows peak detection rules used by a preferred embodiment ofthe invention;

FIG. 24 b shows symbols to be used for interpolation on hexagonal scanpatterns;

FIG. 25 is a flow chart illustrating the fine scan step of FIG. 19;

FIG. 26 is a flow chart illustrating the procedure for performing finescans;

FIG. 27 is a flow chart illustrating the procedure of FIG. 26 forperforming fine scans in the x-y degrees of freedom;

FIG. 28 is an illustration of a fine x-y scan pattern;

FIG. 29 is a flow chart of the hill climbing step of FIG. 26;

FIG. 30 is a flow chart of the plus direction step of FIG. 29; and

FIG. 31 is a flow chart illustrating how model granularity is selectedbased on ratings Q_(g).

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 is a high-level block diagram of one embodiment of the invention.A training image 100 containing an example of a pattern 105 to belocated and/or inspected is presented. A training step 110 analyzes thetraining image and produces a model 120 for subsequent use. At least oneruntime image 130 is presented, each such image containing zero or moreinstances of patterns 135 similar in shape to the training pattern 105.

For each run-time image a run-time step 140 analyzes the image 130,using the model 120, and the list of generalized-DOFs 150. As a resultof the analysis, the run-time step produces a list 160 of zero or moreresults, each result corresponding to an instance of the trained pattern105 in image 130 and containing a pose that maps pattern points tocorresponding image points, and individual probe rating information forinspection purposes.

FIG. 2 shows a preferred embodiment of image processing steps used bythe invention during training step 110 and run-time step 140 forgranularity control, gradient estimation, and boundary detection. Thesesteps process a source image 200, which can be either a training imageor a run-time image.

A low-pass filter 210 and image sub-sampler 220 are used to controlgranularity by attenuating fine detail, such as noise or texture, in thesource image that for a variety of reasons we wish to ignore. Methodsfor low-pass filtering and sub-sampling of digital images are well knownin the art. In a preferred embodiment, a constant-time second-orderapproximately parabolic filter is used, as described in detail in U.S.Pat. No. 6,457,032, entitled “Efficient, Flexible Digital Filtering”.The parabolic filter 210 has a single parameter s, not shown in thefigure, that controls the response of the filter in both dimensions. Ina preferred embodiment, a sub-sampler is used that can produce outputpixels corresponding to non-integer positions of the input image bymeans of the well-known method of bilinear interpolation. Thesub-sampler 220 is controlled by a single parameter g corresponding tothe distance in pixels in both dimensions between the points of theinput image to which the pixels of the output image should correspond.For example if g is 2.5 then output pixels correspond to input points(0,0), (2.5,0), (5,0), (0,2.5), (2.5,2.5), (2.5,5), (5,0), (5,2.5),(5,5), etc. Input points with non-integer coordinates are interpolatedbetween the four surrounding pixels using bilinear interpolation.

For a preferred embodiment, the filter parameter s is tied to thesub-sample parameter g by the following formula:s=round(2.15(g−1))  (2)

Thus there is a single independent parameter g for control of the filter210 and sub-sampler 220, which together constitute the steps used forgranularity control, and therefore granularity is defined to be thevalue g, which is in units of source image pixels. Note that the minimumvalue of g is 1 pixel, which setting effectively disables the filter(since s is 0) and sub-sampler (since the spacing of output pixels isthe same as input pixels), and which corresponds to the granularitylimit imposed by the sensor.

The filtered, sub-sampled image is processed by a gradient estimationstep 230 to produce an estimate of the x (horizontal) and y (vertical)components of image gradient at each pixel. A Cartesian-to-polarconversion step 240 converts the x and y components of gradient tomagnitude and direction. Methods for gradient estimation andCartesian-to-polar conversion are well-known in the art. In a preferredembodiment, the methods described in U.S. Pat. No. 6,408,109, entitled“Apparatus and Method for Detecting and Sub-Pixel Location of Edges in aDigital Image”, herein incorporated by reference, are used. In apreferred embodiment, the source image 200 has 8 bits of gray-scale perpixel. The low-pass filter 210 produces a 16-bit image, taking advantageof the inherent noise-reduction properties of a low-pass filter. Thegradient estimation step 230 uses the well-known Sobel kernels andoperates on either a 16-bit filtered image, if the parameter s is setgreater than 0 so as to enable the filter, or an 8-bit unfiltered imageif the parameter s is set to 0 so as to disable the filter. The x and ycomponents of gradient are always calculated to 16 bits to avoid loss ofprecision, and the gradient magnitude and direction are calculated to atleast 6 bits using the well-known CORDIC algorithm.

The output of Cartesian-to-polar conversion step 240 is a gradientmagnitude image 242 and a gradient direction image 244. These images aresuitable for use by the run-time step 140, but further processing isrequired to identify boundary points for the training step 110. Methodsfor identifying points along image boundaries are well-known in the art.Any such method can be used for practice of the invention, whether basedon gradient estimation or other techniques. In a preferred embodimentshown in FIG. 2, the methods described in detail in U.S. Pat. No.6,408,109, entitled “Apparatus and Method for Detecting and Sub-PixelLocation of Edges in a Digital Image”, herein incorporated by reference,are used. A peak detection step 250 identifies points where the gradientmagnitude exceeds a noise threshold and is a local maximum along a1-dimensional profile that lies in approximately the gradient direction,and produces a list of the grid coordinates (row and column number),gradient magnitude, and gradient direction for each such point. Asub-pixel interpolation step 260 interpolates the position of maximumgradient magnitude along said 1-dimensional profile to determinereal-valued (to some precision) coordinates (x_(i), y_(i)) of the point.The result is a boundary point list 270 of points that lie alongboundaries in the image, which includes the coordinates, direction, andmagnitude of each point.

FIG. 3 shows a flow chart of a preferred embodiment of the modeltraining step 110. A training image 100 containing an example of apattern to be located and/or inspected is analyzed by a series of stepsresulting in a model 120 containing a list of probes. Additional resultsof the training step are that a granularity value g is determined thatis used both during the training step 110 and run-time step 140. Also,the pattern contrast is determined and stored in the model.

In the preferred embodiment shown in FIG. 3, a step 300 selects anappropriate value for the granularity control g and stores the value inthe model 120. Generally lower values of g result in higher accuracy andhigher values of g result in higher speed, so the tradeoff isapplication dependent. For any given pattern, there are limits beyondwhich the granularity may be too large (coarse) to resolve key patternfeatures or too small (fine) to attenuate inconsistent detail such assurface texture. In a preferred embodiment a user can enter a suitablegranularity manually by viewing a display of boundary point list 270superimposed on training image 100. In another preferred embodiment, thevalue g is set automatically based on an analysis of the training image100 as described below.

As shown in FIG. 3 a boundary point detection step 310 processes thetraining image 100 to produce a list of points 270 that lie alongboundaries in the training image. This step is shown in detail in FIG. 2and described above. A connect step 320 connects boundary points toneighboring boundary points that have consistent directions, using rulesfurther described below, to form chains by associating with eachboundary point links to left and right neighbors along the boundaries,if any. A chain step 330 scans the connected boundary points to identifyand catalog discrete chains. For each chain, the starting and endingpoints, length, total gradient magnitude, and whether the chain is openor closed is determined and stored.

A filter step 340 deletes weak chains. A variety of criteria can be usedto identify weak chains. In a preferred embodiment, chains whose totalgradient magnitude or average gradient magnitude are below somespecified parameter are considered weak.

A segment step 350 divides chains into zones of low curvature calledsegments, separated by zones of high curvature called corners. Eachboundary point is marked as a part of a segment or corner. Curvature canbe determined by a variety of methods; in a preferred embodiment, aboundary point is considered a corner if its direction differs from thatof either neighbor by more than 22.5 degrees.

A probe spacing step 360 analyzes the segments found by step 350 anddetermines a probe spacing that would result in all of the segmentsbeing covered by a predetermined target number of probes that aredistributed evenly along the segments. The probe spacing is not allowedto fall beyond certain predetermined limits. In a preferred embodiment,the target number of probes is 64, the lower limit of probe spacing is0.5 pixels, and the upper limit of probe spacing is 4.0 pixels.

A probe generation step 370 creates probes evenly spaced along thesegments found by step 350, and stores them in the model 120.

A contrast step 380 determines the contrast of the pattern 105 in thetraining image 100 by using the run-time step 140 to obtain a resultcorresponding to the pattern 105, and extracting the contrast value fromsaid result. Contrast is defined below.

FIG. 4 shows an example of a portion of a boundary point list 270 for asmall subset of a training image 100 as might be produced by step 310.The boundary points are shown superimposed on a grid 410, which is aportion of the pixel grid of the image input to gradient estimation step230 from which the boundary points were extracted. In a preferredembodiment using the steps of FIG. 2, as further described in U.S. Pat.No. 6,408,109, entitled “Apparatus and Method for Detecting andSub-Pixel Location of Edges in a Digital Image”, herein incorporated byreference, no more than one boundary point will fall within any givengrid element, and there will be no gaps in the boundary due to gridquantization effects. For example, the boundary point 400 falls withingrid element 420, shaded gray in FIG. 4. The boundary point 400 hasgradient direction and magnitude shown by vector 440. Also shown is asmall straight-line section of pattern boundary 460 corresponding to theexample boundary point 400 and normal to the gradient direction 440.This section of boundary is shown primarily to aid in understanding thefigure. Its orientation and position along the gradient direction aresignificant, but its length is essentially arbitrary.

FIG. 5 shows details of the connect step 320 of the training step 110.FIG. 5 a shows the same grid 410, the same example boundary pointsincluding example 400 with gradient 440, and the same example gridelement 420, shaded light gray, as was shown in FIG. 4.

For every boundary point, the grid 410 is examined to identifyneighboring grid elements that contain boundary points to which theboundary point should be connected. For the example boundary point 400in grid element 420, the neighboring grid elements 500 are shown, shadedmedium gray. The neighboring grid elements 500 are examined in two stepsof four neighboring grid elements each, each step in a particular order,determined by the gradient direction 440 of the boundary point 400corresponding to grid element 420.

In one step a left neighbor grid element 510 is identified, and a leftlink 515 is associated with the boundary point 400 identifying theboundary point 517 contained by grid element 510 as its left neighbor.In the other step a right neighbor grid element 520 is identified, and aright link 525 is associated with the boundary point 400 identifying theboundary point 527 contained by grid element 520 as its right neighbor.If a given neighbor cannot be found, a null link is associated. Notethat “left” and “right” are defined arbitrarily but consistently by animaginary observer looking along the gradient direction.

FIG. 5 b shows the order in which neighboring grid elements are examinedfor a boundary point whose gradient direction falls between arrows 540and 542, corresponding to a boundary tangent that falls between dottedlines 544 and 546. The sequence for identifying the left neighbor is +1,+2, +3, and +4. The first neighbor in said sequence that contains aboundary point, if any, is the left neighbor. Similarly, the sequencefor identifying the right neighbor is −1, −2, −3, and −4.

FIG. 5 c shows another example, where the gradient direction fallsbetween arrows 560 and 562, corresponding to a boundary tangent thatfalls between dotted lines 564 and 566. The sequences of neighbors areas shown. The sequences for all other gradient directions are simplyrotations of the two cases of FIGS. 5 b and 5 c.

Note that the sequences given in FIGS. 5 b and 5 c show a preference fororthogonal neighbors over diagonal neighbors, even when diagonalneighbors are “closer” to the direction of the boundary tangent. Thispreference insures that the chains will properly follow a stair-steppattern for boundaries not aligned with the grid axes. Clearly thispreference is somewhat dependent on the specific details of how theboundary point detection step 310 chooses points along the boundary.

Once left and right links have been associated with all boundary points(some of said links may be null), a consistency check is performed.Specifically, the right neighbor of a boundary point's left neighborshould be the boundary point itself, and the left neighbor of a boundarypoint's right neighbor should also be the boundary point itself. If anylinks are found for which these conditions do not hold, those links arebroken by replacing them with a null link. At the end of the connectstep 320, only consistent links remain.

Many alternate methods can be used to establish boundaries within thespirit of the invention.

FIG. 6 shows details of the segment step 350, the probe spacing step360, and the probe generation step 370. In the figure, boundary points600, drawn as diamonds within shaded grid elements, have been marked bysegment step 350 as belonging to corners, and boundary points 620, 622,624, 626, and 640, drawn as circles within unshaded grid elements, havebeen marked as belonging to segments. The boundary points 620, 622, 624,and 626 belong to one segment to be explained in more detail, and theboundary points 640 belong to another segment and are shown forillustration and not further discussed. All of the boundary points ofFIG. 6 are connected by left and right links as shown, for example 660,and form a portion of one chain.

Segment step 350 also determines an arc position for each boundary pointalong a segment, starting with 0 at the left end of the segment andincreasing to the right by an amount equal to the distance between theboundary points along the segment. For example, boundary point 620 is atthe left end of a segment and is at arc position 0.00 as shown. Theright neighbor 622 of boundary point 620 is 1.10 pixels distant and istherefore at arc position 1.10. Similarly the right neighbor 624 ofboundary point 622 is 1.15 pixels distant and is therefore at arcposition 2.25. Finally, the right-most boundary point 626 along thissegment is at arc position 3.20. The total arc length of a segment isdefined to be the arc position of the right-most boundary point, in thisexample 3.20 pixels. Note that the distance between boundary pointsalong a chain can be substantially larger or smaller than 1 pixel,particularly along diagonal boundaries where the boundary points tend tofollow a stair-step pattern. By approximating true arc distance alongthe chain as described above, instead of considering the boundary pointsto be evenly spaced, grid quantization effects are substantiallyreduced.

Probe spacing step 360 determines a spacing value that would result in apredetermined target number of probes evenly distributed among thesegments. The number of probes n that can be fit along a segment of arclength/at a probe spacing of s is given by the formula:

$\begin{matrix}{n = {{{floor}( \frac{l}{s} )} + 1}} & (3)\end{matrix}$

In the example of FIG. 6 where the arc length is 3.2, one probe can fitif the spacing is greater than 3.2, two probes can fit if the spacing isgreater than 1.6 but not greater than 3.2, and so on. Fewer than thetarget number of probes will be used to keep the spacing from fallingbelow a predetermined lower limit, and more than the target number ofprobes will be used to keep the spacing from exceeding a predeterminedupper limit.

Once a probe spacing value has been chosen by probe spacing step 360,the set of probes to be included in the model 120 are generated by probegeneration step 370. In the example of FIG. 6, a probe spacing of 2.5has been chosen, and applying equation 3 shows that two probes will fitalong the example segment. The two probes are centered along thesegment, so that the first probe location 680 is at arc position 0.35,and the second probe location 685 is at arc position 2.85. The positionand direction of each probe are interpolated between the surroundingboundary points. In the example of FIG. 6, the first probe at arcposition 0.35 is placed along the line segment connecting boundarypoints 620 and 622 and at a distance of 0.35 pixels from boundary point620. The gradient direction of the first probe is proportionally0.35/1.10 between the directions of boundary points 620 and 622.Similarly, the second probe at arc position 2.85 is placed along theline segment connecting boundary points 624 and 626 and at a distance of0.60 pixels from boundary point 624. The gradient direction of thesecond probe is proportionally 0.60/0.95 between the directions ofboundary points 624 and 626.

Probe positions and directions may be represented in any convenientcoordinate system, which is referred to in this specification as patterncoordinates.

FIG. 7 shows the set of probes to be included in model 120, resultingfrom training step 110 shown in FIG. 1 and detailed in FIG. 3, for thepattern 105 of FIG. 1. The pattern 105 consists of 2 boundaries, theoutside circular boundary 700 and the inside cross-shaped boundary 720.The boundary 700 results in one closed chain of boundary pointscontaining one segment and no corners, since the curvature of theboundary at each point is sufficiently low. 32 probes, for example 710,are distributed evenly along boundary 700. The boundary 720 results inone closed chain of boundary points containing 12 segments and 12corners. 32 probes, for example 730, are distributed evenly among the 12segments of boundary 720. All probes in FIG. 7 have weight 1.0.

In some applications it is desirable to create a model 120 based on ageometric description of a known shape, rather than by example from atraining image 100. This is referred to as synthetic training, and canbe performed by a human designer or suitable software working from ageometric description such as CAD data. One advantage of synthetictraining is that the model can be made essentially perfect, whereas anyspecific example of an object appearing in a training image may havedefects and since the process of extracting boundary information, evenfrom a high quality example, is subject to noise, grid quantizationeffects, and other undesirable artifacts. Another advantage of synthetictraining by human designers is that the model designer can useapplication-specific knowledge and judgment to design the probes,including use of variable weights and negative weights, to achievehigher detection reliability.

FIG. 8 shows a human-designed model for locating or inspecting a roundedrectangle pattern using the invention. An ideal rounded rectangle 800 tobe synthetically trained has boundary 805. 36 probes are placed alongthe boundary 805 as shown. 12 probes including examples 812 and 816 areplaced along the top of the boundary 805, and 12 probes includingexamples 810 and 814 are placed along the bottom of the boundary 805.Each of these 24 probes placed along the top and bottom have weight 1.0,shown by the relative lengths of the probes. 20 of these 24 probes,including examples 810 and 812, point straight up or down, and the other4, including examples 814 and 816, are rotated slightly due to theexpected rounding of the corners.

6 probes including examples 822 and 826 are placed along the left of theboundary 805, and 6 probes including examples 820 and 824 are placedalong the right of the boundary 805. Each of these 12 probes placedalong the left and right have weight 2.0, shown by the relative lengthsof the probes. 8 of these 12 probes, including examples 820 and 822,point straight left or right, and the other 4, including examples 824and 826, are rotated slightly due to the expected rounding of thecorners.

The use of weight 1.0 for the top and bottom, and weight 2.0 for theleft and right, makes the total influence of each edge the same. Thesame effect could be achieved by adding 6 more probes each to the leftand right sides, packing them together more densely, but this wouldresult in 33% more probes to be processed for what may be little or noimprovement in the quality of information extracted from the image. Theuse of variable positive weights allows the designer the flexibility tomake that tradeoff.

It can be seen that any sufficiently long straight boundary in therun-time image will match almost perfectly any of the four sides of themodel, even though such a straight boundary is not part of a rectangle.Similarly, any sufficiently long right-angle boundary in the run-timeimage will match almost perfectly fully one-half of the model, eventhough such a boundary is not part of a rectangle. This matching betweenthe model and image features that are not part of the pattern to belocated, which is substantial in this case due to the geometricsimplicity of the pattern, significantly reduces the ability of themodel to discriminate between true instances of the pattern and otherimage features. The problem is made more severe if the application callsfor the model to be used over a wide range of angles. The problem can bemitigated by the use of negative weight probes.

In FIG. 8 imaginary lines 830, 832, 834, and 836 are drawn extendingfrom the edges of the rounded rectangle 800. Probes with weights of−2.0, drawn with dashed lines and open arrow heads, and includingexamples 840, 842, 844, and 846, are placed along the imaginary lines asshown. Example negative weight probes 840, 842, 844, and 846 are placedalong imaginary line 830, 832, 834, and 836 respectively. With thisarrangement, any match function used by the invention that considersnegative weight probes, such as the second and third match functionsdescribed in the summary section, will score approximately 0 against asufficiently long straight boundary, and approximately 0.25 instead of0.5 against a sufficiently long right-angle boundary.

Generalizing this example, one can see that a suitable arrangement ofpositive probes surrounded by negative probes along a line can be usedto discriminate line segments from lines and rays. Furthermore, asuitable arrangement of positive probes bounded by negative probes onone side, along a line, can be used to discriminate rays from lines.

In the following paragraphs and associated figures describing in detailrun-time step 140, certain typographic and symbolic conventions are usedthat are set forth in FIG. 9. Descriptions of named sets of data arepresented in tabular format 900. A named set of data is analogous to arecord, structure, or class as used by many conventional programminglanguages familiar to those skilled in the art. Such a set consists of acollection of named elements of various types. One or more instances ofany given set may be used in a specific embodiment of the invention, asappropriate. A specific instance of a named set is called an object, inaccord with the conventions of object-oriented programming, which usagecan be distinguished by context from physical or simulated objects to belocated by the invention. In the example table 900, the name of the setis given in box 905 as shown, and each element of the set occupies a rowof the table 900. In each row the name of the element is given in column910, the type of the element in column 915, and a descriptive summary ofthe element in column 920. Element names generally follow theconventions of common high-level programming languages for lexicalprogram elements commonly referred to as identifiers. When an element'stype is given as a real number, or as containing real numbers, theelement is assumed to take on non-integral values and be represented inany specific embodiment by a fixed or floating point number of suitablebut unspecified precision. When an element's type is given as a functionof certain argument types and producing a certain value type, theelement is assumed to be a function pointer or virtual member functionas defined by common programming languages such as C and C++. The readershould understand that these descriptions of implementation are intendedto be descriptive and not limiting, and that many suitable alternativeimplementations can be found within the spirit of the invention.

When flowcharts are used to express a sequence of steps, certainconventions are followed. A rectangle, for example 930, indicates anaction be performed. A diamond, for example 940, indicates a decision tobe made and will have two labeled arrows to be followed depending on theoutcome of the decision.

A right-pointing pentagon, for example 950, indicates a that a sequenceof loop steps are to be executed for each of a sequence of values ofsome variable. The loop steps are found by following an arrow, e.g. 952,extending from the point of the pentagon, and when the loop steps havebeen executed for the entire sequence of values, flow is to continue byfollowing an arrow, e.g. 954, extending from the bottom of the pentagon.

An oval, for example 960, indicates a termination of a sequence of loopsteps or a termination of a procedure. A down-pointing pentagon, forexample 970, indicates a connection to a like-named pentagon on anotherfigure.

At times a description in a flow chart or other text can be made moreclearly and concisely in a form similar to that used in conventionalprogramming languages. In these cases a pseudo-code will be used, whichwhile not following precisely the syntax of any specific language willnevertheless be easily understood by anyone skilled in the art.Pseudo-code is always written in the bold, fixed-spaced font illustratedin rectangle 930. Pseudo-code is generally used for variables,references to elements of objects, simple arithmetic expressions, andprocedure calls. Often the syntactic detail of programming languagesobscures rather than reveals the workings of the invention, and in thesecases English descriptions will be used instead of pseudo-code, forexample as shown in diamond 940. In such cases implementation detailscan be filled in by those skilled in the art. In still other cases,particularly for vector and matrix operations, greatest clarity can beachieved using standard mathematical notation, which can representcompactly many arithmetic operations; the use of detailed pseudo-code orEnglish would only obscure the intent.

FIG. 10 shows a data set that represents a model 120. Element probes1000 is a list of probes created by training step 370. Elementgranularity 1010 is the granularity value g chosen during training step300 and used during training step 310 to obtain the boundary points.Element contrast 1020 is the pattern contrast measured during trainingstep 380.

FIG. 11 a shows a data set 1190 that represents a probe. Elementposition 1100 is a 2-vector that specifies the position of the probe inpattern coordinates. Element direction 1110 specifies the gradientdirection expected by the probe, again relative to the probe coordinatesystem. In a preferred embodiment, a binary angle is used to representdirection to simplify angle arithmetic using well-known methods. Elementweight 1120 specifies the weight assigned to the probe, which can bepositive or negative. In a preferred embodiment, zero weight probes arenot used.

FIG. 11 b shows a data set 1195 that represents a compiled probe. A listof compiled probes is generated from the probe list 1000 stored in model120 based on a specific combination of generalized-DOF parameters thatspecify an overall coordinate transform representing the non-translationdegrees of freedom. Element offset 1130 is the pixel address offset ofthe probe in the gradient magnitude image 242 and the gradient directionimage 244 (i.e. the same offset applies to both images). Elementdirection 1140 is the expected gradient direction, mapped to imagecoordinates. Element weight 1150 is the probe weight, copied from thecorresponding probe object and in some embodiments converted from a realto a scaled integer to take advantage of integer multiply hardware.

FIGS. 12 a and 12 b gives details of a function compileProbes 1200 thatconverts the list of probe objects 1000 stored in model 120 to a list ofcompiled-probe objects 1195. Starting on FIG. 12 a, functioncompileProbes 1200 takes map input 1202 and probeMER input 1204 asshown, and returns a list of compiled-probe objects. Note that probeMERinput 1204 is a reference to a rectangle to be set by compileProbes, andso may be considered an output or side-effect, but is listed as an inputfollowing the usual programming convention that a function can returnonly one value or object.

Step 1205 and step 1210 perform initialization as shown. Loop step 1215specifies a sequence of steps for each probe object in the list of probeobjects 1000 stored in model 120. Step 1220 sets up a new compiled-probeobject. Step 1225 copies the weight 1150 from the corresponding probeobject, and in some embodiments may convert it to a format more suitedto efficient processing on the available hardware. Continuing to FIG. 12b, in step 1230, some definitions are made for the subsequent math. Instep 1235, the probe position 1100 is mapped to image coordinates usingthe input coordinate transform 1202 and rounded to an integer imagepixel offset in x (e.g., horizontal) and y (e.g., vertical), from whichis computed an offset 1130 based on the address difference of pixels inthe gradient magnitude image 242 and the gradient direction image 244.The use of a single offset value 1130, instead of the (x, y) pair,allows higher speed access to gradient magnitude or direction as the setof compiled probes are translated around the images.

In step 1240 the expected gradient direction 1110 is mapped to an imagecoordinate relative value 1140. The formula for the mapped direction1140 effectively does the following, reading the vector and matrixoperations 1242 right to left:

-   -   Construct a unit vector 1270 in the gradient direction, with        respect to pattern coordinates, by computing the cosine and sine        of the angle.    -   Rotate the unit vector 90° 1272 to get a direction along the        boundary that contains the probe.    -   Map the rotated unit vector to image coordinates 1274 to get a        boundary direction in image coordinates.    -   Rotate the mapped rotated unit vector −90° 1276 to get a        direction normal to the boundary in image coordinates.    -   If the determinant of the transform matrix is negative, the        transform changes the left-handedness or right-handedness of the        coordinate system, so rotate the vector 180° 1278 because the        −90° of the previous step should have been +90°.    -   Compute the angle of the resulting vector 1280 using the        well-known version of the arctangent function of two arguments        whose result is in the range 0° to 360°.

Note that in computing mapped expected gradient direction 1140 theboundary direction is mapped instead of the gradient direction. This isnecessary to handle the general case where the transform matrix C is notorthonormal. If C is orthonormal, i.e. if C₁₁=C₂₂ and C₁₂=−C₂₁, thenstep 1240 can be replaced with a step that simply adds the constantarctan(C₂₁/C₁₁) to the probe direction 1110.

Note as shown in step 1240 that these calculations can be simplifiedconsiderably 1290. In a preferred embodiment the arctangent function iscomputed using the well-known CORDIC method.

Step 1245 keeps track of the minimum enclosing rectangle in integerpixels of the mapped probes, for subsequent use by the invention todetect duplicate results by determining the extent to which pairs ofresults overlap. Note that the size and shape of the minimum enclosingrectangle will vary depending on the settings of the generalized-DOFparameters, and so must be recomputed for each such setting.

Step 1250 marks the end of the loop steps; control flows back to step1215 on FIG. 12 a to continue with the next probe. If there are no moreprobes, control flows to step 1255, which returns the list ofcompiled-probe objects to the caller of the compileProbes function 1200.

FIG. 13 gives details for the match functions used by the invention.FIGS. 13 a and 13 b show examples of direction rating factor functions.A direction rating factor is value between 0 and 1 that indicates degreeof match between a probe's expected gradient direction 1140 and theactual gradient direction found in a gradient direction image 244 underthe probe. A direction rating factor function produces a directionrating factor as a function of direction error, defined as an anglemeasured from the expected gradient direction to the actual gradientdirection. Any of a variety of direction rating factor functions couldin principal be used to practice the invention.

There are two general types of direction rating factor functions, calledconsider polarity and ignore polarity functions. The difference betweenthe two types is in how they handle direction errors at and around 180°,which corresponds to a gradient direction opposite from what wasexpected, implying that the boundary is in the expected orientation butthe dark-to-light transition of image brightness across the boundary isopposite in polarity from expected. The “consider polarity” functionsreturn 0 at and around 180°, so that polarity reversals do not match thepattern, while the ignore polarity functions return 1 at and around180°, so that polarity reversals do match the pattern. Choice between“consider polarity” and “ignore polarity” is application dependent, andso in a preferred embodiment, the user can select either type.

In a preferred embodiment the consider polarity direction rating factorfunction of FIG. 13 a is used. The function is at 1 from 0° 1300 to11.25° 1302, then falls in a straight line to 0 at 22.5° 1304, remainsat 0 until 337.5° 1306, rises in a straight line to 1 at 348.75° 1308,and remains at 1 until 360° 1310 (which is the same as the 0° point1300). In a preferred embodiment the corresponding ignore polaritydirection rating factor function of FIG. 13 b is used. The points 1320,1322, 1324, 1326, 1328, and 1330 correspond exactly to the points 1300,1302, 1304, 1306, 1308, and 1310, respectively, of FIG. 13 a. The points1332, 1334, 1336, and 1338 correspond to points 1328,1322, 1324, and1326, respectively, but shifted 180°. Note that the points 1320 and 1330have no corresponding points shifted 180°, since these points are anartifact of the decision to start the drawing at 0°.

FIG. 13 c shows an example of a magnitude rating factor function. Amagnitude rating factor is a value between 0 and 1 that indicates adegree of confidence that a particular pixel position lies along aboundary and therefore that a probe test made at said position wouldresult in reliable evidence for, in the case of positive weight probes,or against, in the case of negative weight probes, the existence of aninstance of the trained pattern 105 at the pose under test. A magnituderating factor function produces a magnitude rating factor as a functionof gradient magnitude. Any of a variety of magnitude rating factorfunctions could in principal be used to practice the invention.

In a preferred embodiment the magnitude rating factor function of FIG.13 c is used. The function is 0 at magnitude 0 1350, rises in a straightline to i at a point 1352 corresponding to a certain target magnitudefurther described below, and continues at 1 until the maximum magnitude1354, which in the illustrated embodiment is 255.

The goal in the design of the example magnitude rating factor functionof FIG. 13 c is primarily to distinguish between noise and trueboundaries. The intention is that an embodiment of the invention usingthis magnitude rating factor function be sensitive to the shape ofboundaries but not overly sensitive to the contrast and sharpness of theboundaries. A separate target magnitude point 1352 is computed for eachdistinct pose (placement of probes), since global decisions about whatis noise and what is signal are notoriously unreliable. If we consideronly the set B of positive weight probes with a high direction ratingfactor, it is reasonable to assume that a majority of probes in B liealong a true boundary and not noise. The median gradient magnitudem_(median) under the probes in B is a good guess as to a representativegradient magnitude value corresponding to the boundary. In a preferredembodiment we choose target point 1352 to have the value 0.7 m_(median).

In the following let:

-   -   p_(i) be the offset 1130 of the i^(th) compiled probe 1195;    -   d_(i) be the direction 1140 of the i^(th) compiled probe 1195;    -   w_(i) be the weight 1150 of the i^(th) compiled probe 1195;    -   M(a) be the gradient magnitude at offset a in gradient magnitude        image 242;    -   D(a) be the gradient direction at offset a in gradient magnitude        image 244;    -   R_(dir)( ) be a direction rating factor function, for example        the one in FIG. 13 a or FIG. 13 b; and    -   R_(mag)( ) be a magnitude rating factor function, for example        the one in FIG. 13 c.

With these definitions, it can be seen that for a set of compiled probesplaced at offset a in gradient magnitude image 242 or gradient directionimage 244,

-   -   M(a+p_(i)) is the gradient magnitude under compiled probe i    -   D(a+p_(i)) is the gradient direction under compiled probe i    -   D(a+p_(i))−d_(i) is the direction error at compiled probe i

In the following equations, a term of the form “x=y” or “x>y” is 1 ifthe expression is true and 0 otherwise, following the conventions of theC programming language. This is not standard algebraic notation, buttends to simplify and clarify the formulas.

To avoid having to set a threshold to decide whether or not a probe is amember of B, a probe's direction rating factor is used to specify aweighted membership value. For a specific set of compiled probes (i.e.corresponding to specific settings of the generalized-DOF parameters) aweighted histogram H_(a)(m) of gradient magnitude for complied probesplaced at image offset a is computed as follows:

$\begin{matrix}{{H_{a}(m)} = {\sum\limits_{i}{{\max( {w_{i},0} )}( {{M( {a + p_{i}} )} = m} ){R_{dir}( {{{D( {a + p_{i}} )} - d_{i}}} )}}}} & ( {4a} )\end{matrix}$

Equation 4a states that each bin m of the histogram H is the sum ofdirection rating factors, weighted by probe weight w_(i), for allpositive weight probes where the gradient magnitude under the probe ism. From histogram H can be computed the median gradient magnitude byfinding the value m_(median) such that:

$\begin{matrix}{{\sum\limits_{i = 0}^{m_{median}}{H_{a}(i)}} = {\sum\limits_{i = m_{median}}^{255}{H_{a}(i)}}} & ( {4b} )\end{matrix}$

The first match function S_(1a), used in the coarse scan step 1925 ofFIG. 19, can now be defined as follows:

$\begin{matrix}{{S_{1a}(a)} = \frac{\sum\limits_{i}{{\max( {w_{i},0} )}{R_{dir}( {{{D( {a + p_{i}} )} - d_{i}}} )}}}{\sum\limits_{i}{\max( {w_{i},0} )}}} & ( {5a} )\end{matrix}$

This gives the first match score at any offset a of the compiled probes.As can be seen from equation 5a, only positive weight probes are used.The first variation S_(1b), which, to achieve higher execution speeddoesn't use probe weight (except to select positive weight probes) is:

$\begin{matrix}{{S_{1b}(a)} = \frac{\sum\limits_{i}{( {w_{i} > 0} ){R_{dir}( {{{D( {a + P_{i}} )} - d_{i}}} )}}}{\sum\limits_{i}( {w_{i} > 0} )}} & ( {5b} )\end{matrix}$

The first match function using the first and second variations S₁, whichsubtracts the expected value of the direction rating factor on randomnoise, and is used in a preferred embodiment, is:

$\begin{matrix}{{S_{1}(a)} = \frac{\sum\limits_{i}{( {w_{i} > 0} )\lbrack {{R_{dir}( {{{D( {a + P_{i}} )} - d_{i}}} )} - N} \rbrack}}{( {1 - N} ){\sum\limits_{i}( {w_{i} > 0} )}}} & ( {5c} )\end{matrix}$

where noise term N is given by:

$\begin{matrix}{N = {\frac{1}{360}{\int_{0}^{360}{{R_{dir}(\theta)}\ {\mathbb{d}\theta}}}}} & (6)\end{matrix}$

Using the noise term in the first match function is important because,due to the higher computational cost, gradient magnitude is not used tofilter out noise. Note that the computation of S₁ as specified byequation 5c can be arranged so that the use of N adds no per-probe cost.For the preferred “consider polarity” direction rating factor functionof FIG. 13 a, N= 3/32. For the preferred “ignore polarity” directionrating factor function of FIG. 13 b, N= 3/16.

The second match function S₂ used in the fine scan step 1940 of FIG. 19is:

$\begin{matrix}{{S_{2}(a)} = {\sum\limits_{i}{w_{i}{M( {a + p_{i}} )}{R_{dir}( {{{D( {a + p_{i}} )} - d_{i}}} )}}}} & (7)\end{matrix}$

The third match function S₃ used in the scoring steps 1930 and 1945 is:

$\begin{matrix}{{S_{3}(a)} = \frac{\sum\limits_{i}{w_{i}{R_{mag}( {M( {a + p_{i}} )} )}{R_{dir}( {{{D( {a + p_{i}} )} - d_{i}}} )}}}{\sum\limits_{i}w_{i}}} & (8)\end{matrix}$

FIG. 14 shows a data set 1490, that represents a generalized-DOF. In apreferred embodiment using the C++ programming language, FIG. 14describes an abstract base class that specifies the interface for anygeneralized-DOF. Specific generalized-DOFs, corresponding in a preferredembodiment to concrete derived classes, will be described below. In FIG.14, elements low 1400 and high 1405 specify the range of parametervalues to be searched, as appropriate for the application. If low=high,the parameter value is fixed for this generalized-DOF—no searching isdone, but the fixed parameter value contributes to all poses consideredby run-time step 140 and returned in list of results 160. In a preferredembodiment the invention requires low≦high except for cyclicgeneralized-DOFs.

Element maxStepSize 1410 specifies the maximum allowable increment inparameter value for the coarse scan step 1925 or fine scan step 1940. Ina preferred embodiment maxStepSize 1410 is chosen automatically for eachgeneralized-DOF based on the geometry of the probes, as described below.Element maxStepSize 1410 should be set so that the pattern 105 willmatch sufficiently well against an instance in the run-time image 130even if the pose is off by up to one-half maxStepSize 1410 in everygeneralized-DOF that does not have a fixed parameter value.

Element dupRange 1415 specifies a range of parameter values within whichdistinct results may be considered duplicates. Two results areduplicates if they overlap sufficiently in position (i.e. thetranslation degrees of freedom) and if for each generalized-DOF theirrespective parameter values are within the dupRange 1415 of thatgeneralized-DOF.

Element start 1420 specifies the actual start of the range of parametervalues to be searched, which extends half stepSize 1435 beyond therequested range given by low 1400 and high 1405 so that interpolationcan be performed up to the limit of the requested range.

Element numCoarseSteps 1430 gives the number of steps in stepSize 1435increments to be used during coarse scan step 1925. Element stepSize1435 is derived from maxStepSize 1410 and the requested range from low1400 to high 1405 such that stepSize 1435 is not greater thanmaxStepSize 1410 and there are an integral number of steps to cover arange that extends one-half step beyond low 1400 and high 1405. Notethat if a generalized-DOF is cyclic and the range covers the entirecycle, then 2 fewer steps are needed because the ends of the range arecoincident.

Element cycle 1440 specifies one cycle for cyclic generalized-DOFs(e.g., 360°), or 0 for non-cyclic generalized-DOFs. Adding orsubtracting cycle 1440 to any parameter value has no effect on the pose.The element cycle 1440 allows cyclic and non-cyclic generalized-DOFs,which are much more similar than different, to share a large body ofcode with only minor special case handling in several places.

Element mapper 1445 is a function that converts a parameter value to anequivalent coordinate transform. In a preferred embodiment, mapper 1445is a virtual function. Element mapper 1445 is the key to thegeneralized-DOF method, because the resulting coordinate transforms canbe composed to produce a pose regardless of the type, number, and orderof generalized-DOFs used by the invention in any given embodiment. In apreferred embodiment the coordinate transform produced by mapper 1445includes a translation vector, but it is always 0.

Elements stepSizeMatrix 1450 and stepSizeFactor 1455 are used to computemaxStepSize 1410 based on the geometry of the probes and the nature ofthe generalized-DOF, as further described below. Element scaleFactor1460, a virtual function in a preferred embodiment, computes the factorby which the pattern is scaled by this generalized-DOF (changed in size)at the middle of the search range between low 1400 and high 1405. Thisis used as a rough estimate of the change in scale from the trainingimage 100 to the run-time image 130, so that certain parameters, such asgranularity 1010, can be adjusted.

FIG. 15 is a table that details specific generalized-DOFs that can beused with the invention. Many other variations not shown can be devisedbased on the teachings disclosed herein. Each row of the table describesa specific generalized-DOF, while the columns generally specify valuesfor specific elements.

Column 1500 describes the parameter used by the generalized-DOF. In apreferred embodiment, rotational generalized-DOFs, e.g. 1540 and 1545,use an angle parameter in degrees. Radians are not used because onecycle (i.e. 2π) cannot be represented exactly in any practical device.Size-related generalized-DOFs use either a scale factor parameter, e.g.1570, 1575, and 1580, or a logarithmic scale factor parameter, e.g.1550, 1555, and 1560. Aspect ratio generalized-DOFs use either a ratioparameter, e.g. 1585, or a logarithmic ratio parameter, e.g. 1565.

Element cycle 1440 is set to 360° for rotational generalized-DOFs, e.g.1540 and 1545, and 0 otherwise. Element stepSizeFactor 1455 is set to180/π to convert radians to degrees for rotational generalized-DOFs,e.g. 1540 and 1545, and 1 otherwise. Function element scaleFactor 1460returns the scale factor at the geometric midpoint of the search rangefor uniform size-related generalized-DOFs, e.g. 1550 and 1570, thesquare root of the scale factor at the geometric midpoint of the searchrange for non-uniform size-related generalized-DOFs, e.g. 1555, 1560,1575, and 1580, and 1 for non-size related generalized-DOFs. Fornon-uniform size-related generalized-DOFs, the square root—is areasonable estimate of the overall effect when size is varying in onedimension but not the other.

For function element mapper 1445, only the 2×2 matrix component of thecoordinate transform is shown; the vector translation component is 0.Element stepSizeMatrix 1450 is determined by taking the derivative

$\frac{\mathbb{d}\;{{mapper}(x)}}{\mathbb{d}x}$and evaluating the resulting matrix at parameter value x such thatmapper(x) is the identity transform.

FIG. 16 shows details of the list of generalized-DOFs 150 used in apreferred embodiment of the invention. As described in the summarysection, the list 150 specifies nested loops for the coarse scan step1925 and fine scan step 1940. The list 150 specifies both the nestingorder for scanning the search space and the order in which thecoordinate transforms produced by the mapper functions 1445 are composedto get the overall pose for the non-translation degrees of freedom. Forthe preferred embodiment shown in FIG. 16, a log y size generalized-DOF1560 is the first element of list 150, is the outermost loop in thescanning sequence, and its transform is applied first in mapping frompattern to image coordinates. Next is log x size generalized-DOF 1555,followed by log size 1550, and finally rotation 1540, which is theinnermost loop in the scanning sequence, and its transform is appliedlast in mapping from pattern to image coordinates. Other orders arepossible, and are chosen to suit the particular application.

There is a redundant degree of freedom among list 150 elements log ysize 1560, log x size 1555, and log size 1550. These threegeneralized-DOFs cover only a two degree of freedom search space, whichany of the three possible pairs are sufficient to cover. The use ofthese three, however, gives the user of the invention much greaterflexibility in specifying the search space than if only twonon-redundant generalized-DOFs were used. Specifically, the user has 7sensible choices-holding all three fixed, allowing anyone to vary, andallowing any pair to vary.

FIG. 17 shows a data set that represents a result corresponding to aninstance of a pattern 105 in a run-time image 130. A list of results 160is the primary output of the invention. Element position 1700 specifiesthe position in image coordinates of the origin of the patterncoordinate system at the match pose, i.e. the pose corresponding to theinstance of pattern 105 in run-time image 130 represented by the result.Element probeMER 1710 specifies the minimum enclosing rectangle in imagecoordinates of the probes at the match pose, and is used to determinewhether or not two results overlap sufficiently in position to beconsidered possible duplicates.

Element score 1720 is the match score, which is refined and updated asrun-time step 140 progresses. During coarse scan step 1925 it is set toa value interpolated among a set of values of first match function S₁(a)of Equation 5c, evaluated at a corresponding set of offsets a ingradient direction image 244 that includes a local maximum of S₁ and itsneighbors. Subsequently during the coarse scan step 1925, element score1720 is refined by interpolating among neighboring results in eachnon-fixed generalized-DOF. During fine scan step 1940 it is set tosecond match function S₂ of Equation 7. During steps 1930 and 1945,score 1720 is set to the value of third match function S₃ of Equation 8.

Element contrast 1730 is set to the median gradient magnitude valuem_(median), as defined by equations 4a and 4b, and computed as part ofdetermining target gradient magnitude point 1352 needed for third matchfunction S₃.

Element DOFParameters 1740 is a list of generalized-DOF parameterscorresponding to generalized-DOF list 150 and specifying thenon-translation degrees of freedom of the pose represented by theresult. Element DOFIndices 1750 is a list of step indices correspondingto generalized-DOF list 150. A step index for a generalized-DOF is aninteger between 0 and numCoarseSteps −1 that indicates a specific stepduring the coarse scan step 1925 for said generalized-DOF. ElementDOFIndices 1750 is used to identify results that are neighbors along ageneralized-DOF, as further described below.

FIG. 18 shows how position overlap is calculated for a pair of resultsto determine if they might be neighbors or duplicates. Overlap is avalue between 0 and 1 inclusive that indicates degree of positionoverlap (i.e. overlap in the translation degrees of freedom) between tworesults.

In FIG. 18 a, rectangle 1800 is the element probeMER 1710 of a firstresult, with center point 1802 at distance w₁ 1804 from the left andright edge and distance h₁ 1806 from the top and bottom edge. Similarly,rectangle 1810 is the element probeMER 1710 of a second result, withcenter point 1812 at distance w₂ 1814 from the left and right edge anddistance h₂ 1816 from the top and bottom edge.

Since in general the minimum enclosing rectangles are of differentshapes, the relative positions d_(x) 1818 and d_(y) 1808 of the centerpoints 1802 and 1812 as shown in FIG. 18 b are used instead of area ofintersection to determine overlap. The formula for overlap is theproduct of an x overlap term and a y overlap term, as follows:

$\begin{matrix}{\lbrack {\max( {{1 - \frac{d_{x}}{w_{1} + w_{2}}},0} )} \rbrack \cdot \lbrack {\max( {{1 - \frac{d_{y}}{h_{1} + h_{2}}},0} )} \rbrack} & (9)\end{matrix}$

FIG. 18 c shows examples of overlap 1.0 1820, overlap 0.5 1825 and 1835,overlap 0.25 1840, and overlap 0.0 1830 and 1845.

FIG. 19 is a top-level flow chart of a preferred embodiment of run-timestep 140. Step 1900 sets pseudo-code identifiers model and dofList tomodel 120 and list of generalized-DOFs 150, respectively, for referenceby subsequent pseudo-code.

Step 1905 determines the “nominal” scale factor s_(nominal) of the setof ranges of parameters of all the generalized-DOFs. This is a crudeestimate of the typical scale factor of the non-translation posesgenerated by list of generalized-DOFs 150, and is most useful when therange of scale factors is small and significantly different from 1.0,i.e. the patterns are expected to be significantly larger or smaller inrun-time image 130 than training image 100. The value s_(nominal) is theproduct, over generalized-DOF elements 1490 of dofList 150, of allvalues scaleFactor 1460 of FIG. 14.

Step 1910 computes run-time granularity g_(run) as the product oftraining granularity model·granularity 1010 and s_(nominal) but not lessthan 1.0. Step 1915 processes run-time image 130 to obtain gradientmagnitude image 242 and gradient direction image 244, following thesteps of FIG. 2, and using run-time granularity g_(run).

Step 1920 determines, for each generalized-DOF element 1490 of dofList150, settings for maxStepSize 1410, start 1420, numCoarseSteps 1430, andstepSize 1435.

In a preferred embodiment, maxStepSize 1410 is computed from thegeometry of the probes and the number of non-fixed generalized-DOFs. Inthe following, let:

-   -   p_(i) be the position vector 1100 of the i^(th) probe 1190;    -   (x_(i), y_(i)) be the components of the position vector 1100 of        the i^(th) probe 1190;    -   θ_(i) be the direction 1110 of the i^(th) probe 1190;    -   u_(i) be a unit vector in direction 1110 of the i^(th) probe        1190;    -   w_(i) be the weight 1120 of the i^(th) probe 1190;    -   n be the number of generalized-DOFs on dofList 150 that are not        fixed, i.e. where low 1400 does not equal high 1405;    -   M be a stepSizeMatrix 1450 of a generalized-DOF 1490; and    -   f be a stepSizeFactor 1455 of a generalized-DOF 1490.

Define the center of projection c=(c_(x), c_(y)) of a list of probes1000 as the point that minimizes the sum of squared distance betweenlines, passing through the probes positions normal to the gradientdirection, and said point. The center of projection is similar to centerof mass of a set of points, except that center of projection considersthe probes to provide information in only one degree of freedom, thegradient direction, instead of 2 degrees of freedom as for a normalpoint. Center of projection can be computed as follows:

$\begin{matrix}{{r_{i} = {{x_{i}{\cos( \theta_{i} )}} + {y_{i}{\sin( \theta_{i} )}}}}{c_{x} = \frac{{\sum\limits_{i}{r_{i}{\cos( \theta_{i} )}{\sum\limits_{i}{\sin^{2}( \theta_{i} )}}}} - {\sum\limits_{i}{r_{i}{\sin( \theta_{i} )}{\sum\limits_{i}{{\cos( \theta_{i} )}{\sin( \theta_{i} )}}}}}}{{\sum\limits_{i}{{\cos^{2}( \theta_{i} )}{\sum\limits_{i}{\sin^{2}( \theta_{i} )}}}} - ( {\sum\limits_{i}{{\cos( \theta_{i} )}{\sin( \theta_{i} )}}} )^{2}}}{c_{y} = \frac{{\sum\limits_{i}{r_{i}{\sin( \theta_{i} )}{\sum\limits_{i}{\cos^{2}( \theta_{i} )}}}} - {\sum\limits_{i}{r_{i}{\cos( \theta_{i} )}{\sum\limits_{i}{{\cos( \theta_{i} )}{\sin( \theta_{i} )}}}}}}{{\sum\limits_{i}{{\cos^{2}( \theta_{i} )}{\sum\limits_{i}{\sin^{2}( \theta_{i} )}}}} - ( {\sum\limits_{i}{{\cos( \theta_{i} )}{\sin( \theta_{i} )}}} )^{2}}}} & (10)\end{matrix}$

For each generalized-DOF element 1490 of dofList 150, maxStepSize 1410is computed as follows:

$\begin{matrix}{b = \sqrt{\frac{\sum\limits_{i}{w_{i} \cdot \lbrack {u_{i} \cdot ( {M( {p_{i} - c} )} )} \rbrack^{2}}}{\sum\limits_{i}w_{i}}}} & (11) \\{{maxStepSize} = \frac{f \cdot {\min( {\frac{1.5}{b},0.2} )}}{\sqrt{n}}} & (12)\end{matrix}$

Equation 11 computes a baseline b in units of distance that is a measureof the sensitivity of the probes to motion induced by thegeneralized-DOFs parameter. For example, a circular boundary would haveprobes pointing radially that would be very sensitive to size changesbut insensitive to rotation. In the equation, probe position vectors p1100, relative to center of projection c, are adjusted depending on thespecific generalized-DOF by matrix M 1450, and then the dot product withunit vector u in the probe direction 1110 is taken. In the example of acircular boundary, for a size generalized-DOF, e.g. 1550 or 1570, M issuch that the said dot product gives the radius of the circle, andbaseline b also becomes the radius of the circle. For a rotationgeneralized-DOF such as 1540, M is such that the said dot product gives0. In equation 12, the bigger the baseline b, the more sensitive is thepattern to changes in the parameter of the generalized-DOF, and so thesmaller the step size should be. The constants 1.5 and 0.2 are used in apreferred embodiment, although other constants can be used to obtainsimilar performance in other applications. The step size is furtherreduced by the square root of the number of generalized-DOFs that canvary, since if the pose can be off in more degrees of freedomsimultaneously then the step sizes must be smaller.

Once maxStepSize 1410 is set, elements start 1420, numCoarseSteps 1430,and stepSize 1435 are set as shown in FIG. 20 and described below.

Step 1925 does the coarse scan of the entire search space, producing apreliminary list of results 160 for further processing. Note that thesecond argument I is the identity transform. Coarse scan step 1925 isdescribed in more detail below.

Step 1930 evaluates the third match function S₃ (equation 8) for eachelement of results 160, at the pose determined by coarse scan step 1925.The purpose of step 1930 is to qualify each result as being sufficientlyhigh in score to be worth running the fine scan step 1940. Step 1930 isreasonably fast since only one pose is evaluated for each result.

Step 1935 discards both weak results and duplicate results. In apreferred embodiment, a weak result is one whose score 1720 is belowsome fraction of a global accept threshold chosen to be suitable for theapplication. For step 1935, said fraction is 0.75—only resultssubstantially weaker than the accept threshold are discarded, since thefine scan step 1940 might improve the score. In a preferred embodiment apair of results are considered duplicates if their overlap value, asdescribed in FIG. 18, is at least 0.8, and if their lists ofgeneralized-DOF parameters DOFParameters 1740 agree to within dupRange1415 for all generalized-DOFs on dofList 150. For all duplicate pairs inresults 160, the member of the pair with the lower score 1720 isdiscarded.

Step 1940 does the fine scan on each remaining result in results 160, asfurther described below. Step 1940 establishes the final position 1700,probeMER 1710, and DOFParameters 1740 for each result 1790.

Step 1945 evaluates the third match function S₃ (equation 8) for eachelement of results 160, at the pose determined by fine scan step 1940.Step 1945 establishes the final score 1720 and contrast 1730 for eachresult 1790. Step 1945 also stores individual probe ratings, which arethe product of magnitude rating factor R_(mag) and direction ratingfactor R_(dir) from equation 8, in result element 1760.

Step 1950 repeats step 1935 to discard weak and duplicate results. Forstep 1950, however, weak results are defined to be those whose score1720 is below 0.9 of the global accept threshold.

FIG. 20 provides more details on the setting of generalized-DOF elementsstart 1420, numCoarseSteps 1430, and stepSize 1435 in step 1920. Step2000 determines if the requested range of parameter values between low1400 and high 1405 is sufficiently large to require coarse scanning. Ifthe range from low 1400 to high 1405 is not greater than maxStepSize1410, then no coarse scanning is required and so numCoarseSteps 1430 isset to 1 in step 2005, and start 1420 is set to the midpoint of therange in step 2010.

If coarse scanning is required, numCoarseSteps 1430 is set in step 2020to be the range divided by maxStepSize 1410, but rounded up to thenearest integer. Note that this is not yet the final value fornumCoarseSteps, because boundary conditions must be considered. Theactual step size stepSize 1435 is then set in step 2025 to be the rangedivided by numCoarseSteps. The result of steps 2020 and 2025 is thatstepSize 1435 is the smallest value that can cover the range in the sameintegral number of steps as can be done by maxStepSize 1410.

Step 2030 tests to see if the generalized-DOF is cyclic and if therequested range covers the full cycle. If so, start 1420 is set to low1400 in step 2040—in this case it doesn't really matter where the scanstarts. The value of numCoarseSteps 1430 computed in step 2020 iscorrect, because the range has no end points. If the requested range isnot a full cycle (including non-cyclic generalized-DOFs), start 1420 isset one-half step below low 1400 in step 2050, and numCoarseSteps 1430is increased by two in step 2060 to cover the end points.

FIG. 21 a is a flow chart of a function coarseScanDOF 2100 that scansall of the generalizedDOFs on input list dofList 2102, and returns alist of results 1790 describing poses representing possible instances ofpattern 105 in run-time image 130. Function coarseScanDOF 2100 is arecursive function—it operates on an outermost generalized-DOF which isthe first element of input list 2102, and calls itself to operate on theinner generalized-DOFs represented by the rest of the list. At eachlevel of the recursion a partial pose is constructed by composing acurrent mapper 1445 transform with the input map 2104 constructed byrecursion levels representing outer generalized-DOFs, and passing saidpartial pose along to recursion levels representing innergeneralized-DOFs. At the outermost level in step 1925 the identitytransform is provided for input map 2104. At the innermost level, whenstep 2110 determines that input dofList 2102 is null, thenon-translation portion of the pose is complete and the procedurecoarseScanXY 2200 of FIG. 22 a is called in step 2112 to scan thetranslation degrees of freedom.

If input dofList 2102 is not null, step 2114 extracts the first elementrepresenting the current generalized-DOF, and the rest of the listrepresenting inner generalized-DOFs. An empty results list is allocatedin step 2116. Loop step 2118 executes search loop 2130 for a sequence ofvalues of a step index. Each iteration of loop search 2130 scans oneparameter value of the current generalized-DOF. When the scanning iscomplete, loop step 2120 executes peak loop 2160 for every result foundby scan loop 2118. Each iteration of peak loop 2160 determines whether aresult is a peak—a local maximum in the current generalized-DOF—and ifso, interpolates it, otherwise marks it for deletion. Step 2122 actuallydeletes all results marked for deletion, and finally step 2124 returnsthe list of remaining results.

FIG. 21 b is a flow chart of search loop 2130. Step 2140 computes theparameter value corresponding to the current step index. Step 2142 isthe recursive call to coarseScanDOF 2100 that scans the innergeneralized-DOFs. Note the second argument, which composes the currentmapper transform with map input 2104. Loop steps 2144, 2146, 2148, and2150 add the current step index and parameter value to the beginning ofelements DOFIndices 1750 and DOFParameters 1740 in every result 1790returned by recursive step 2142. Finally, in step 2151 the list ofresults returned by recursive step 2142 is added to the end of resultsand in step 2154 the search loop continues to the next index at step2118 of FIG. 21 a.

FIG. 21 c is a flow chart of peak loop 2160, which operates on anspecific result r. Steps 2170 and 2172 search results for a previous andnext neighbor of r, respectively. A neighbor of r is a result whose stepindex for the current generalized-DOF, which is the first element ofDOFIndices 1750, differs by exactly 1 from that of r, whose step indicesfor all inner generalized-DOFs, which are the second and subsequentelements if any of DOFIndices 1750, differs by no more than 1 from thatof r, and whose overlap value with r (equation 9) is at least 0.8. Forcyclic generalized-DOFs where the scan range covers the full cycle, stepindex differences are considered modulo numCoarseSteps 1430. A previousneighbor is a neighbor where the current step index difference is −1,and a next neighbor is a neighbor where the current step indexdifference is +1.

Step 2174 determines if result r is a peak (local maximum in score)compared to its neighbors. Note that if a previous or next neighbor wasnot found, its score is assumed to be 0. If not, r is marked fordeletion in step 2176. If so, the parameter value is interpolated instep 2178, and, if both a previous and next neighbor were found, thescore is interpolated in step 2180, and then the loop is continued instep 2182.

The following 3-point parabolic interpolation functions are used:

$\begin{matrix}{{{{InterpPos}( {l,c,r} )} = \frac{r - l}{{4c} - {2( {l + r} )}}}{{{InterpPos}( {l,c,r} )} = {c + \frac{( {r - l} )^{2}}{{16c} - {8( {l + r} )}}}}} & (13)\end{matrix}$

The interpolated parameter value is obtained by adding:

-   -   stepSize*InterpPos(prevResult.score, r.score, nextResult.score        to the current value. The interpolated score is:        -   InterpScore(prevResult.score, r.score, nextResult.score)

FIG. 22 a is a flow chart of a function coarseScanXY 2200 used bycoarseScanDOF 2100 to scan the translation degrees of freedom. In takesone input map 2202 that specifies the non-translation portion of thepose, and returns a list of results 1790 representing potentialinstances of pattern 105 in run-time image 130 at poses corresponding tomap 2202.

Step 2210 allocates an empty list of results. Step 2212 compiles list ofprobes 1000 using function compileProbes 1200 resulting in a list ofcompiled probes 1195 and a minimum enclosing rectangle of the probes.Step 2214 evaluates first match function S₁(a) at a subset of possibleoffsets a in gradient magnitude image 244, as selected by a scan patterndescribed below. Only image offsets a such that all of the compiledprobes, each placed at its offset 1130 relative to a, contained in saidimage 244, are evaluated by step 2214.

Loop step 2216 iterates over the scores evaluated in step 2214, and step2218 examines the scores and looks for local maxima above a noisethreshold. In a preferred embodiment, the noise threshold is set to 0.66of the global accept threshold. Detection of local maxima is describedbelow. When a local maximum above the noise threshold is found, newresult loop 2240 is executed; otherwise, control flows to step 2220 andthen back to 2216. When all the scores have been examined by step 2218,control passes to step 2222 which returns any results found.

FIG. 22 b is a flow chart of new result loop 2240, which is executedwhenever a local maximum in score above the noise threshold is found bystep 2218. Step 2242 allocates a new result 1790. Step 2244 initializesthe values by setting position 1700 to an interpolated position of themaximum score, score 1720 to an interpolated score, probeMER 1710 to theminimum enclosing rectangle computed by compileProbes 1200, offset bythe interpolated position of the maximum score, and lists DOFParameters1740 and DOFIndices 1750 to empty lists.

Step 2246 searches the results found so far for a duplicate of the newresult. In a preferred embodiment, a duplicate is a result with overlapvalue (equation 9) of at least 0.8. Steps 2248 and 2250 select amongthree cases. If no duplicate was found, step 2252 adds the new result tothe list. If a duplicate was found with a score lower than the newresult, step 2254 replaces the duplicate with the new result. If aduplicate was found with a score not lower than the new result, step2256 discards the new result. Finally, step 2258 transfers control backto step 2216 to continue looking for local maxima.

FIG. 23 shows coarse x-y scan patterns used by step 2214 in a preferredembodiment. In each example the dots indicate relative positions to beevaluated. Traditionally template matching systems have evaluated amatch score at every position, or in a square pattern of sub-sampledpositions, and the same may be done in a less preferred embodiment ofthe invention. The patterns shown in FIG. 23, called hexagonal scanpatterns due to the shape of the neighborhoods, e.g. 2304, 2314, 2324,and 2334, are both more efficient and more flexible than a square or anyother pattern method. With hexagonal patterns it is possible to evaluatea fraction of possible positions, i.e. ½ for example 2300, ¼ for example2310, ⅙ for example 2320, and 1/9 for example 2330, that is notrestricted to reciprocals of perfect squares, as for square subsamplingpatterns, and not significantly anisotropic, as for rectangularsub-sampling patterns. For a given fraction of positions, the worst casedistance from any point in the plane to the nearest evaluated point isless for the hexagonal pattern than for any other pattern. Since thegrid itself is square it is only possible to approximate a hexagonalpattern, but the worst case distance is still very close to optimum. Ina preferred embodiment, the ¼ pattern 2310 is used.

FIG. 23 shows example evaluated points 2302, 2312, 2322, and 2332, andcorresponding neighborhoods 2304, 2314, 2324, and 2334 for use by peakdetection step 2218 and interpolation step 2244.

FIG. 24 a shows peak detection rules used by a preferred embodiment forstep 2218. Evaluated point 2400, with neighborhood 2405 correspondingschematically to any hexagonal neighborhood, e.g., 2304, 2314, 2324, and2334, is considered a local maximum if its score is greater than orequal to the scores of neighbors 2410, 2415, and 2420, and greater thanthe scores of neighbors 2425, 2430, and 2435.

FIG. 24 b gives symbols to be used for interpolation on hexagonal scanpatterns, as is used in step 2244. Evaluated score at local maximum z2440, with neighborhood 2445 corresponding schematically to anyhexagonal neighborhood, e.g., 2304, 2314, 2324, and 2334, hasneighboring scores in a first grid direction x_(p) 2450 and x_(n) 2465,in a second grid direction u_(p) 2455 and u_(n) 2470, and a third griddirection v_(p) 2460 and v_(n) 2475. For each grid direction athree-point parabolic interpolation is computed as follows:r _(x)=InterpPos(x _(n) ,z,x _(p))r _(u)=InterpPos(u _(n) ,z,u _(p))  (14)r _(v)=InterpPos(v _(n) ,z,v _(p))

Construct lines 2480, 2482, and 2484 normal to grid directions x, u, andv respectively, and at a distance r_(x), r_(u), and r _(y), respectivelyfrom local maximum 2440 in the direction of x_(p) 2450, u_(p) 2455, andv_(p) 2460 respectively. The interpolated position 2490 is the pointthat minimizes the sum squared distance between lines 2480, 2482, and2484 and point 2490. In the example, r_(x) and r_(u) are negative andr_(v) is positive. The offset (Δx, Δy) of interpolated point 2490 fromthe local maximum point 2440 is given by:

$\begin{matrix}{\begin{pmatrix}{\Delta\; x} \\{\Delta\; y}\end{pmatrix} = {J\begin{pmatrix}r_{x} \\r_{u} \\r_{v}\end{pmatrix}}} & (15)\end{matrix}$

where 2×3 matrix J is:

Pattern J 1/2 2300 $\frac{1}{2}\begin{pmatrix}2 & 1 & {- 1} \\0 & 1 & 1\end{pmatrix}$ 1/4 2310 $\frac{1}{35}\begin{pmatrix}50 & 25 & {- 25} \\0 & 42 & 42\end{pmatrix}$ 1/6 2320 $\frac{1}{22}\begin{pmatrix}39 & 24 & {- 15} \\{- 3} & 32 & 35\end{pmatrix}$ 1/9 2330 $\frac{1}{96}\begin{pmatrix}207 & 117 & {- 90} \\{- 21} & 169 & 190\end{pmatrix}$

The interpolated score is:

max(InterpScore(x_(n), z, x_(p)), InterpScore (u_(n), z, u_(p)),InterpScore (v_(n), z, v_(p))).

In a less preferred embodiment using a hexagonal scan pattern,interpolation is accomplished by fitting an elliptical paraboloid toscores 2440, 2450, 2455, 2460, 2465, 2470, 2475, and defining theposition and height of the extremum of said elliptical paraboloid to bethe interpolated position and score.

FIG. 25 is a top level flow chart of fine scan step 1940. Loop step 2500does some number of iterations of refinement, where each iterationrefines all of the results and the step size is halved each time. In apreferred embodiment, two iterations are done. Loop steps 2510 and 2520halve the step size for each generalized-DOF on dofList 150. Loop steps2530 and 2540 call procedure fineScanDOF 2600 for each result. Note thatthe second argument I to fineScanDOF 2600 is the identity transform.

FIG. 26 is a flow chart of procedure fineScanDOF 2600, with inputsdofList 2602, map 2604, and result 2606. Function fineScanDOF 2600 is arecursive function—it operates on an outermost generalized-DOF which isthe first element of input list 2602, and calls itself to operate on theinner generalized-DOFs represented by the rest of the list. At eachlevel of the recursion a partial pose is constructed by composing acurrent mapper 1445 transform with the input map 2604 constructed byrecursion levels representing outer generalized-DOFs, and passing saidpartial pose along to recursion levels representing innergeneralized-DOFs. At the outermost level in step 2540 the identitytransform is provided for input map 2604. At the innermost level, whenstep 2610 determines that input dofList 2602 is null, thenon-translation portion of the pose is complete and control flows tofine scan x-y step 2700 to scan the translation degrees of freedom.

If input dofList 2602 is not null, step 2620 extracts the first elementrepresenting the current generalized-DOF, and the rest of the listrepresenting inner generalized-DOFs. Step 2630 fetches the current valueof the parameter corresponding to the current generalized-DOF from listDOFParameters 1740 in the result 2606 being refined. Step 2640 callsfineScanDOF 2600 recursively to scan the inner generalized-DOFs forposes corresponding to the current parameter setting of the currentgeneralized-DOF. Step 2650 tests to determine if the currentgeneralizedDOF is fixed, and if so step 2660 returns, otherwise controlflows to hill climb step 2900.

FIG. 27 is a flow chart of the translation portion of the fine scan step1940. Step 2710 makes a list of compiled probes based on thenon-translation degrees of freedom specified by input map 2604. Step2720 evaluates second match function S₂ at a set of offsets surroundingthe current position 1700 of result, determined by fine scan pattern2800. Step 2730 stores the highest match score found in step 2720 inelement score 1720 of result. Step 2740 sets position 1700 of result tothe position of the highest match score, interpolated betweennorth-south (y) neighbors, and east-west (x) neighbors, using InterpPos(equation 13). Step 2750 returns from procedure fineScanDOF 2600.

FIG. 28 shows the fine scan pattern 2800 used in a preferred embodiment.32 offsets are evaluated surrounding the current best position 1700.

FIG. 29 is a flow chart of hill climb step 2900, used for non-fixedgeneralized-DOFs. In step 2910 two temporary results rp (plus direction)and rn (minus direction) are allocated and initialized to input result2606. Step 2920 evaluates poses at plus and minus one step size from thecurrent parameter value. Step 2930 tests to see if the score improves ineither the plus or minus direction. If not, control flows to interpolatestep 2950. If so, step 2940 tests to see which direction is better, plusor minus. If the plus direction is better (has higher score), controlflows to plus direction step 3000. If the minus direction is better,control flows to minus direction step 2990. Once plus direction step3000 or minus direction step 2990 finishes, step 2950 interpolates theparameter value by adding

-   -   stepSize*InterpPos(rn.score, result.score, rp.score)        to the appropriate parameter value in list DOFParameters 1740 of        result 1790.

FIG. 30 is a flow chart of plus direction step 3000. Step 3010 tests tosee if stepping in the plus direction would exceed the high limit 1405of the search range. If so, control flows to ending step 3060. If not,step 3020 steps the generalized-DOF parameter in the plus direction.Step 3030 shifts the temporary results over by one step, and step 3040evaluates a new set of poses in the plus direction. Step 3050 tests tosee if the score at the new parameter value is greater than the previousbest score, and if so control flows back to step 3010 to continuescanning in the plus direction. If not, scanning terminates at step3060, and control flows back to step 2950.

Minus direction step 2990 is identical in form to plus direction step3000, with obvious modifications to scan in the other direction. Detailsare not shown.

In a preferred embodiment, granularity 1010 is chosen automaticallyduring training step 300 based on an analysis of pattern 105. Theportion of training image 100 corresponding to pattern 105 is processedto extract boundary points 270 at a set of granularities according toFIG. 2, and each resulting boundary point list 270 is processedaccording to FIGS. 4, 5, and 6, corresponding to steps 320, 330, 340,and 350, to produce, for each boundary point, a left and right neighbor,to produce chains, to discard weak chains, and to determine the totalarc length of each chain.

In the following, for any given boundary point list 270 and associatedchains at some granularity g, let:

-   -   p_(r) be the position vector of the r^(th) boundary point.    -   x_(r), x_(r) be the components of p_(r).    -   d_(r) be the gradient direction of the r^(th) boundary point.    -   M_(r) be the gradient magnitude of the r^(th) boundary point.    -   u_(r) be the unit vector in the gradient direction d_(r).    -   a be the area of pattern 105 in pixels.    -   l_(r) be the arc length of the chain, which is the sum of the        arc lengths of all the chain segments, containing the r^(th)        boundary point.

An estimate of a suitable granularity g_(est) is made using the formula:

$\begin{matrix}{g_{est} = \sqrt{\frac{\sqrt{a}}{8}}} & (16)\end{matrix}$

A set of integer granularities in the range 1 to g_(max), inclusive, isanalyzed, whereg _(max)=floor(√{square root over (2)}·g _(est))  (17)

For each granularity g in the above range, an overall rating Q_(g) iscomputed. The formulas for the rating Q_(g) have the following goals:

-   -   To prefer granularities closer to g_(est).    -   To prefer that boundary points be spread out, covering more        area.    -   To prefer longer chains.    -   To prefer smaller curvature along the chains.    -   To prefer stronger gradient magnitudes.    -   To prefer that boundary points not be near other parallel        boundaries.    -   To normalize the rating so that ratings at different        granularities can be compared.

Define a curvature rating function of neighboring boundary points r andj as follows:

$\begin{matrix}{{C( {r,j} )} = {1 - {\min\lbrack \frac{\max( {{{{d_{r} - d_{j}}}_{360} - 16.875^{\circ}},0} )}{11.25^{\circ}} \rbrack}}} & (18)\end{matrix}$

In this formula, the absolute difference of the two gradient directionsis taken module 360°, so that the result is positive and in the range0-180°. The curvature rating is 1 for direction differences less than16.875°, 0 for direction differences above 28.125°, and proportionallybetween 0 and 1 for differences between 16.875° and 28.125°.

Define a parallel magnitude value e_(r) whose purpose is to estimate thegradient magnitude of boundaries close to and parallel to the r^(th)boundary point. Let G_(r) be a set of boundary points foundapproximately along a line passing through boundary point r and ingradient direction d_(r). Define

$\begin{matrix}{e_{r} = {\sum\limits_{j \in G_{r}}{M_{j} \cdot {P( {r,j} )} \cdot {D( {r,j} )}}}} & (19)\end{matrix}$

so that e_(r) is the sum over all boundary points j in G_(r) of theproduct of gradient magnitude M_(j) a parallel rating P(r,j) and adistance rating D(r,j), where

$\begin{matrix}{{P( {r,j} )} = {1 - {\min\lbrack {\frac{\max( {{{{d_{r} - d_{j}}}_{180} - 11.25^{\circ}},0} )}{11.25^{{^\circ}}},1} \rbrack}}} & (20) \\{{D( {r,j} )} = {1 - {\min\lbrack {\frac{\max( {{{{u_{r} \cdot ( {p_{r} - p_{j}} )}} - 1.0},0} )}{4.0},1} \rbrack}}} & (21)\end{matrix}$

Parallel rating P(r,j) is similar to curvature rating C(r,j), exceptthat the absolute difference of gradient direction is taken module 180°,and ranges from 0° to 90°.

Distance rating D(r,j) is based on the distance between boundary pointsr and j in gradient direction d_(r). This is the effect of the dotproduct shown. Distances smaller than 1 pixel get a rating of 1.0,greater than 5 pixels get a rating of 0, and proportionally in between.

Define a weight W_(r) for the r^(th) boundary point.W _(r) =l _(r) ^(0.25) ·C(r,left)·C(r,right)·max(m _(r) −e _(r),0)

where “left” and “right” identify the left and right neighbors of ther^(th) boundary point along the chain, respectively. The weight W_(r) isthe gradient magnitude m_(r), but discounted for near by parallelmagnitudes e_(r), further discounted for excessive left or rightcurvature ratings, and enhanced based on chain length according to apower law. In a preferred embodiment the power is 0.25.

Now define the overall rating

$\begin{matrix}{Q_{g} = {g^{1.625}e^{- {{\log{(\frac{g}{g_{est}})}}}}I}} & (22)\end{matrix}$

where I is boundary point moment of inertia:

$\begin{matrix}{I = \sqrt{{\sum\limits_{r}{w_{r}x_{r}^{2}}} + {\sum\limits_{r}{w_{r}y_{r}^{2}}} - \frac{( {\sum\limits_{r}{w_{r}x_{r}}} )^{2} + ( {\sum\limits_{r}{w_{r}x_{r}}} )^{2}}{\sum w_{r}}}} & (23)\end{matrix}$

The moment of inertia factor takes into account the spread of theboundary points and their weights, which in turn take into account chainlength, curvature, gradient magnitude, and parallel boundaries. Thefactor

$\begin{matrix}e^{- {{\log{(\frac{g}{g_{est}})}}}} & (24)\end{matrix}$discounts (or attenuates) the moment of inertia based on the ratio of gto the estimated granularity g_(est). The factorg^(1.625)  (25)compensates for the fact that all distances scale by g so that number ofboundary points scales by g, moment of inertia by g², and arc lengthl^(0.25) scales by g^(0.25), for total scale by g^(3.25). The squareroot in the formula for I makes the scale factor g^(1.625).

FIG. 31 is a flow chart showing how model granularity 1010 is selectedbased on ratings Q_(g). In step 3100 a granularity g_(best) and ratingQ_(best) are initialized. Loop step 3105 scans all integer values ofgranularity n in the range 1 to g_(max), inclusive. For each loopinteraction, variables q and g are initialized in step 3110. The looplooks for maxima of Q, interpolates both rating and granularity at themaxima, and then chooses the interpolated granularity at the maximumwith the highest interpolated rating.

Steps 3115, 3120, and 3125 handle the case where n is 1, the smallestgranularity considered. Step 3120 tests to see if n=1 is a maximum, andif so step 3125 is the “interpolation” for this case.

Steps 3130, 3135, and 3140 handle the case where n is the largestgranularity considered. Step 3135 tests for a maximum, and step 3140 isthe “interpolation”.

Steps 3143 and 3145 handle the case where n is neither the smallest norlargest granularity considered, and so a 3-point interpolation can bedone. Step 3143 tests for a maximum. The formulas shown for step 3145implement a 3-point parabolic interpolation similar to InterpPos andInterpScore, except that the domain of the parabola is log granularityinstead of the usual linear scale.

Steps 3150 and 3155 replace Q_(best) and g_(best) with q and g if abetter rating q has been found, and step 3160 continues the loop at step3105.

When the loop is finished, step 3170 sets model granularity 1010 to theinterpolated granularity g_(best) at the maximum with highest ratingQ_(best).

Other modifications and implementations will occur to those skilled inthe art without departing from the spirit and the scope of the inventionas claimed. Accordingly, the above description is not intended to limitthe invention except as indicated in the following claims.

1. In probe-based pattern matching, a method for locating patterns, themethod comprising: providing a database that comprises: a probe-basedpattern model having: a plurality of positive probes at selected pointsalong a boundary of the pattern, at least one imaginary straight segmentparallel to a segment of the boundary of the pattern, and a plurality ofnegative probes at selected points along the imaginary straight segment,each negative probe having a negative weight; and providing a computerprogrammed to perform the step of: using the probe-based pattern model,and probe-based pattern matching, to locate at least one instance of thepattern in an image.
 2. The method of claim 1, wherein the plurality ofpositive probes at selected points along a boundary of the pattern wereplaced according to a method including: using the computer to run aprogram to perform the steps of: determining an arc position for eachboundary point along a segment of the boundary, starting with zero at afirst boundary point at a first end of the segment, and increasing whilemoving away from the first end by an amount equal to the distancebetween the boundary points along the segment; determining the total arclength of the segment as being the arc position of a boundary point mostdistal from the first boundary point of the segment; determining a probespacing value along the segment; and determining a target number ofprobes to be placed along the segment using the total arc length of thesegment, and the probe spacing value.
 3. The method of claim 2, whereindetermining a probe spacing value along the segment includes: selectinga probe spacing value along the segment such that the probe spacingvalue is no less than 0.5 pixels, and such that the probe spacing valueis no more than 4.0 pixels.
 4. The method of claim 2, whereindetermining a target number of probes to be placed along the segmentincludes: computing a ratio of the arc length of the segment to theprobe spacing value along the segment; computing a floor function of theratio; and adding
 1. 5. The method of claim 2, after determining atarget number of probes to be placed along the segment, furthercomprising: using fewer than the target number of probes to ensure thatthe spacing does not fall below a predetermined lower limit.
 6. Themethod of claim 2, after determining a target number of probes to beplaced along the segment, further comprising: using more than the targetnumber of probes to ensure that the spacing does not exceed apredetermined upper limit.
 7. The method of claim 6, wherein a positionof each probe is interpolated from positions of surrounding boundarypoints.
 8. The method of claim 6, wherein a direction of each probe isinterpolated from directions of surrounding boundary points.
 9. Themethod of claim 1, wherein the plurality of positive probes at selectedpoints along a boundary of the pattern were placed according to a methodincluding: using the computer to run a program to perform the steps of:identifying at least one long segments of the boundary of the pattern;identifying at least one short segments of the boundary of the pattern;associating a long positive weight with each positive probe along eachlong segment; associating a short positive weight with each positiveprobe along each short segment; and setting the value of each shortpositive weight as being greater than the value of each long positiveweight.
 10. The method of claim 1, wherein the plurality of positiveprobes at selected points along a boundary of the pattern were placedaccording to a method including: using the computer to run a program toperform the steps of: identifying at least one long segments of theboundary of the pattern; identifying at least one short segments of theboundary of the pattern; and placing more probes per unit length on eachshort segment than on each long segment.
 11. The method of claim 1,wherein the plurality of negative probes at selected points along theimaginary straight segment, each negative probe having a negativeweight, were placed according to a method including: using the computerto run a program to perform the steps of: making the magnitude of eachnegative weight greater than the magnitude of positive weights on the atleast one straight segment that was extended to provide the imaginarystraight segment.
 12. The method of claim 1, wherein the plurality ofnegative probes at selected points along the imaginary straight segment,each negative probe having a negative weight, were placed according to amethod including: using the computer to run a program to perform thesteps of: making the number of negative probes along the imaginarystraight segment exceed the number positive probes on the at least onestraight segment that was extended to provide the imaginary straightsegment.
 13. The method of claim 1, wherein the plurality of positiveprobes at selected points along a boundary of the pattern were placedaccording to a method including: using the computer to run a program toperform the steps of: identifying a pair of long segments; identifying apair of short segments; associating a long positive weight with eachpositive probe along each long segment; associating a short positiveweight with each positive probe along each short segment; and settingthe value of each short positive weight as being greater than the valueof each long positive weight.
 14. The method of claim 1, wherein theplurality of positive probes at selected points along a boundary of thepattern were placed according to a method including: using the computerto run a program to perform the steps of: identifying a pair of longsegments; identifying a pair of short segments; and placing as manyprobes on each long segment as there are probes placed on eachcorresponding short segment.
 15. The method of claim 1, wherein theplurality of negative probes at selected points along the imaginarystraight segment, each negative probe having a negative weight, wereplaced according to a method including: using the computer to run aprogram to perform the steps of: making the magnitude of each negativeweight greater than the magnitude of each positive weights on a straightsegment.
 16. The method of claim 1, wherein the plurality of negativeprobes at selected points along the imaginary straight segment, eachnegative probe having a negative weight, were placed according to amethod including: using the computer to run a program to perform thesteps of: making the number of negative probes along the imaginarystraight segment exceed the number positive probes on a straightsegment.
 17. The method of claim 13, wherein only two pairs of longsegments are identified.
 18. The method of claim 13, wherein only twopairs of short segments are identified.
 19. The method of claim 1,wherein the probe-based pattern model has 36 probes.
 20. The method ofclaim 1, wherein weights of probes of the short segments are twice thevalue of weights of probes of the long segments.
 21. The method of claim10 wherein the step of placing more probes per unit length on each shortsegment than on each long segment include placing as many probes on eachlong segment as there are probes placed on each short segment.
 22. Inprobe-based pattern matching, an apparatus for locating patterns, theapparatus comprising: a database including a probe-based pattern model,the model including: (i) a plurality of positive probes at selectedpoints along a boundary of the pattern; (ii) at least one imaginarystraight segment parallel to a segment of the boundary of the pattern;and (iii) a plurality of negative probes at selected points along theimaginary straight segment, each negative probe having a negativeweight; and a computer readable memory including a program to performthe steps of: using the probe-based pattern model, and probe-basedpattern matching, to locate at least one instance of the pattern in animage.
 23. The apparatus of claim 22 wherein the pattern includes atleast one long segment of the boundary and at least one short segment ofthe boundary of the pattern, a long positive weight associated with eachpositive probe along each long segment and a short positive weightassociated with each positive probe along each short segment, where thevalue of each short positive weight is being greater than the value ofeach long positive weight.
 24. The apparatus of claim 22 wherein thepattern includes at least one long segment of the boundary and at leastone short segment of the boundary of the pattern and the probes includemore probes per unit length on each short segment than on each longsegment.
 25. The apparatus of claim 22, wherein a mimicked-shape, havingpart of the boundary and the at least one imaginary straight segment,match a feature in the image that is not in an instance of the patternin the image.
 26. The apparatus claim 22, wherein the program incomputer readable memory places a plurality of positive probes atselected points along a boundary of the pattern, the program furthercomprising: computing an arc position for each boundary point along asegment of the boundary, starting with zero at a first boundary point ata first end of the segment, and increasing while moving away from thefirst end by an amount equal to the distance between the boundary pointsalong the segment; computing the total arc length of the segment asbeing the arc position of a boundary point most distal from the firstboundary point of the segment; computing a probe spacing value along thesegment; and calculating a target number of probes to be placed alongthe segment using the total arc length of the segment, and the probespacing value.
 27. The apparatus claim 22, wherein the magnitude of eachnegative weight being greater than the magnitude of one of the positiveweights on the at least one straight segment that was extended toprovide the imaginary straight segment.